Nov 02 19:49:09.654177 ip-10-0-130-111 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 19:49:09.654187 ip-10-0-130-111 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 19:49:09.654194 ip-10-0-130-111 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 19:49:09.654481 ip-10-0-130-111 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 19:49:19.665870 ip-10-0-130-111 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 19:49:19.665885 ip-10-0-130-111 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2a5aef47648f4978909a980959a584e2 -- Nov 02 19:51:23.812579 ip-10-0-130-111 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:51:24.173738 ip-10-0-130-111 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:51:24.173738 ip-10-0-130-111 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:51:24.173738 ip-10-0-130-111 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:51:24.173738 ip-10-0-130-111 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:51:24.178699 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178624 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:24.178699 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178698 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178702 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178706 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178709 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178712 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178715 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178717 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178720 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178722 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178725 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178728 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178732 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178735 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178737 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178740 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178742 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178745 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178748 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178751 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178753 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:24.178762 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178756 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178759 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178762 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178765 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178767 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178774 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178779 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178783 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178787 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178789 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178792 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178795 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178797 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178800 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178802 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178805 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178807 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178810 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178813 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:24.179219 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178815 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178817 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178820 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178822 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178825 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178828 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178830 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178833 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178835 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178838 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178840 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178843 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178845 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178848 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178852 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178855 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178858 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178861 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178863 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:24.179665 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178867 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178869 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178872 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178874 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178877 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178879 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178882 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178885 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178889 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178892 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178895 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178898 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178901 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178903 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178906 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178909 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178912 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178914 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178917 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:24.180139 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178920 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178922 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178925 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178927 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178930 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178932 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178935 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178938 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178940 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178943 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178946 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178949 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178951 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178954 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178957 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178960 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178962 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178964 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178967 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178969 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:24.180700 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178972 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178974 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178976 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178979 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178981 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178985 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178987 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.178990 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179752 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179758 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179761 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179764 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179766 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179770 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179772 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179775 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179777 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179780 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179783 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179786 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:24.181184 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179788 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179791 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179794 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179796 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179799 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179802 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179804 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179807 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179810 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179813 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179816 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179818 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179821 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179823 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179826 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179829 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179832 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179834 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179837 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:24.181671 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179839 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179842 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179845 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179848 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179850 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179853 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179856 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179858 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179861 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179863 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179865 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179868 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179870 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179873 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179876 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179878 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179881 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179883 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179886 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179888 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:24.182133 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179891 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179894 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179897 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179899 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179902 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179905 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179907 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179910 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179913 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179916 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179918 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179921 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179923 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179926 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179929 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179931 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179934 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179936 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179941 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:24.182616 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179945 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179949 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179952 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179956 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179960 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179963 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179966 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179968 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179971 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179974 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179977 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179979 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179982 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179984 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179987 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179990 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179993 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179995 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.179997 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:24.183095 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180000 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180003 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180006 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180008 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180011 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180014 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180016 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180019 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180022 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180025 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180028 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180030 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180034 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180036 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180039 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180041 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180044 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180545 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180551 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180554 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:24.183550 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180557 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180560 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180563 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180565 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180568 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180570 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180573 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180576 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180580 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180582 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180585 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180588 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180590 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180593 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180596 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180599 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180602 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180605 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180608 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180610 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:24.184014 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180613 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180615 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180618 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180621 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180623 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180626 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180629 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180631 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180634 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180636 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180639 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180641 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180644 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180647 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180649 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180652 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180655 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180657 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180660 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:24.184493 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180663 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180665 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180668 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180671 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180674 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180676 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180679 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180682 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180685 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180687 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180690 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180693 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180695 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180698 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180701 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180704 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180707 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180709 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180712 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180715 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:24.184936 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180717 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180720 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180722 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180725 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180727 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180730 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180732 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180735 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180737 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180740 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180742 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180745 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180747 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180750 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180752 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180755 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180758 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180760 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180763 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180765 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:24.185415 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180768 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180770 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180773 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180776 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180778 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180780 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180783 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180785 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180788 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180790 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180794 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180798 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180801 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180804 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180806 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180809 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180811 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180814 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180816 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:24.185879 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180819 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180821 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180824 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180829 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.180832 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181922 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181935 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181942 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181947 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181953 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181956 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181961 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181965 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181969 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181972 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181978 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181981 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181985 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181988 2581 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181991 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181994 2581 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.181997 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182000 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:51:24.186323 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182004 2581 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182007 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182010 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182013 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182016 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182020 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182024 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182027 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182031 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182034 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182037 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182040 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182043 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182046 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182051 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182054 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182057 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182060 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182063 2581 flags.go:64] FLAG: --enable-server="true" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182066 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182070 2581 flags.go:64] FLAG: --event-burst="100" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182074 2581 flags.go:64] FLAG: --event-qps="50" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182077 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182080 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:51:24.186880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182084 2581 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182088 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182091 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182095 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182098 2581 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182101 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182104 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182107 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182110 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182113 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182116 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182118 2581 flags.go:64] FLAG: --feature-gates="" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182123 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182126 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182129 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182132 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182136 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182140 2581 flags.go:64] FLAG: --help="false" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182143 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182147 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182150 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182153 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182156 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:51:24.187512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182160 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182163 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182166 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182168 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182171 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182175 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182178 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182181 2581 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182184 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182188 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182191 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182194 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182196 2581 flags.go:64] FLAG: --lock-file="" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182199 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182202 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182205 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182211 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182213 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182216 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182219 2581 flags.go:64] FLAG: --logging-format="text" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182222 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182225 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182228 2581 flags.go:64] FLAG: --manifest-url="" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182231 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182236 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:51:24.188048 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182240 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182244 2581 flags.go:64] FLAG: --max-pods="110" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182247 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182250 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182253 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182256 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182259 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182262 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182265 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182283 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182287 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182290 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182294 2581 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182297 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182300 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182304 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182308 2581 flags.go:64] FLAG: --port="10250" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182312 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182315 2581 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c8c2ef10f7847467" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182318 2581 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182321 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182324 2581 flags.go:64] FLAG: --register-node="true" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182327 2581 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182331 2581 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182333 2581 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182336 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:51:24.188675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182339 2581 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182343 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182347 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182350 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182353 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182356 2581 flags.go:64] FLAG: --runonce="false" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182360 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182363 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182366 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182369 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182372 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182375 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182378 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182381 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182384 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182387 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182390 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182393 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182396 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182399 2581 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182402 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182408 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182412 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182415 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182419 2581 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:51:24.189601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182421 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182424 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182427 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182430 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182433 2581 flags.go:64] FLAG: --v="2" Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182437 2581 flags.go:64] FLAG: --version="false" Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182441 2581 flags.go:64] FLAG: --vmodule="" Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182446 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182449 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182602 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182606 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182609 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182612 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182617 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182620 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182622 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182625 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182628 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182631 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182633 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182636 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182639 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:24.190376 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182641 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182644 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182646 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182649 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182651 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182654 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182657 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182661 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182663 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182666 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182668 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182671 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182674 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182676 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182679 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182681 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182684 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182686 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182689 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182691 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:24.190901 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182694 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182696 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182699 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182702 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182705 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182708 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182711 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182713 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182717 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182721 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182725 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182728 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182731 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182733 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182736 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182739 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182741 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182744 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:24.191384 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182746 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182749 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182752 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182755 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182757 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182760 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182762 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182765 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182767 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182770 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182772 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182775 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182777 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182779 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182782 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182785 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182787 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182790 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182792 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182795 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:24.191795 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182798 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182800 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182803 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182806 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182808 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182811 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182813 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182816 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182818 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182821 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182823 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182826 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182828 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182831 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182834 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182836 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182839 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182842 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182844 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182847 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:24.192332 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182849 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182852 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182854 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182857 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182859 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182861 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182864 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182867 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182869 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182871 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182874 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182877 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182879 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182882 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182884 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.182910 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:51:24.192794 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182944 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182949 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182952 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182955 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182957 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182960 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182962 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182965 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182968 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182971 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182973 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182976 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182979 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182982 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182984 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182987 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182989 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182992 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182994 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:24.193178 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.182997 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183000 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183003 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183007 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183011 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183014 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183018 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183020 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183023 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183026 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183028 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183033 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183035 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183038 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183040 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183043 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183045 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183048 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183050 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:24.193738 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183053 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183055 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183057 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183060 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183064 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183067 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183070 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183072 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183075 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183078 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183080 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183082 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183085 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183087 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183091 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183093 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183096 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183098 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183101 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183104 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:24.194186 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183107 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183109 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183112 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183114 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183118 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183121 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183123 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183126 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183128 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183131 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183134 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183136 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183139 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183141 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183143 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183146 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183151 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183153 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183157 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183160 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:24.194678 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183162 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183165 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183167 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183170 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183172 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183175 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183178 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183180 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183183 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183185 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183188 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183190 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183193 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183195 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183198 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183200 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183204 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183207 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183209 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183212 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:24.195147 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183215 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183218 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183221 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183223 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183225 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183228 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183230 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.183233 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.191613 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.191631 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.191650 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.191655 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191722 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191728 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191731 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191735 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191738 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191741 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191743 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191746 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:24.195633 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191749 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191751 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191754 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191756 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191759 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191761 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191764 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191767 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191774 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191776 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191780 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191782 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191785 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191788 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191790 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191793 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191796 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191799 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191802 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191804 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:24.196113 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191807 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191810 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191813 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191816 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191819 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191821 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191824 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191826 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191829 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191831 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191834 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191837 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191839 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191842 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191844 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191847 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191849 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191852 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191854 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191857 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:24.196649 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191860 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191862 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191865 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191867 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191870 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191873 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191875 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191878 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191880 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191883 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191885 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191888 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191890 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191895 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191898 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191903 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191907 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191911 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191915 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191919 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:24.197100 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191923 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191928 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191931 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191934 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191936 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191939 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191942 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191946 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191950 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191954 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191957 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191960 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191963 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191965 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191968 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191971 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191974 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191976 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191979 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:24.197815 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191982 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191984 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191987 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191990 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191993 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191995 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.191998 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192001 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192003 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192007 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192011 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192014 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192017 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192020 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192023 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192025 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192028 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192030 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:24.198263 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192033 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.192061 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192232 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192237 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192240 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192243 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192246 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192257 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192261 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192263 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192266 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192269 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192288 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192292 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192296 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192298 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:24.198743 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192301 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192304 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192308 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192311 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192315 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192317 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192320 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192322 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192325 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192328 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192331 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192333 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192336 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192339 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192342 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192344 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192347 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192350 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192352 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192355 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:24.199125 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192357 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192360 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192362 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192365 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192368 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192371 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192373 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192376 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192379 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192381 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192384 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192386 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192389 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192392 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192394 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192397 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192400 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192403 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192405 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:24.199640 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192407 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192410 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192412 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192415 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192417 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192420 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192422 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192425 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192427 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192430 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192433 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192435 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192438 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192440 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192443 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192446 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192448 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192451 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192455 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192458 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:24.200107 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192460 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192463 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192466 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192469 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192471 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192474 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192476 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192479 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192482 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192484 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192487 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192489 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192491 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192494 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192496 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192499 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192501 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192504 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192506 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:24.200755 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192509 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192511 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192514 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192517 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192519 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192522 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192524 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192526 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192529 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192532 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192534 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192537 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192539 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192542 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.192566 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:51:24.201197 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192598 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192602 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192605 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192608 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192611 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192614 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192617 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192620 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192622 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192625 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192627 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192630 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192632 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192635 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192638 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192641 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192643 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192646 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192648 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:51:24.201590 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192651 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192654 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192656 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192659 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192661 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192664 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192666 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192669 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192671 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192674 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192676 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192680 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192683 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192685 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192687 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192690 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192692 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192695 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192697 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:51:24.202034 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192700 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192702 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192705 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192709 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192712 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192716 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192719 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192722 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192725 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192729 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192733 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192736 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192739 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192741 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192744 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192746 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192749 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192751 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192754 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:51:24.202586 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192756 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192759 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192761 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192766 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192769 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192772 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192775 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192778 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192780 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192783 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192786 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192788 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192791 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192794 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192796 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192799 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192802 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192805 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192807 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192810 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:51:24.203381 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192812 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192815 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192817 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192820 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192823 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192825 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192828 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192830 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192833 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192835 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192838 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192840 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192843 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192846 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192848 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192852 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192855 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192858 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192860 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192863 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:51:24.203932 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192866 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192869 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192871 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192875 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192877 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192880 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192882 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192885 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:24.192887 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.192968 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.196132 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.197224 2581 server.go:1014] "Starting client certificate rotation" Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.197336 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.197378 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.199555 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:51:24.204515 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.200754 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:51:24.214834 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.214819 2581 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:51:24.218774 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.218758 2581 log.go:25] "Validated CRI v1 image API" Nov 02 19:51:24.220449 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.220427 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:51:24.227642 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.227617 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 def1b433-09c4-4b28-bfd7-bcc798fdae81:/dev/nvme0n1p3 fd7926d3-74c9-4adb-8aab-099fd39106b4:/dev/nvme0n1p4] Nov 02 19:51:24.227694 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.227637 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 19:51:24.233794 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.233674 2581 manager.go:217] Machine: {Timestamp:2026-11-02 19:51:24.231839858 +0000 UTC m=+0.320178758 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3219534 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20cac3cf9745bcfeaf4e7f2f2d4771 SystemUUID:ec20cac3-cf97-45bc-feaf-4e7f2f2d4771 BootID:2a5aef47-648f-4978-909a-980959a584e2 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c5:e5:bd:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c5:e5:bd:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:d3:58:a0:29:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:51:24.233794 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.233788 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:51:24.233932 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.233875 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:51:24.235193 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.235168 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:51:24.235349 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.235196 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-111.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:51:24.235397 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.235352 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:51:24.235397 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.235360 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:51:24.235449 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.235399 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:51:24.235449 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.235417 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:51:24.236741 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.236731 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:51:24.241717 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.241705 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:51:24.241836 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.241827 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:51:24.242664 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.242655 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:51:24.242696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.242669 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:51:24.242696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.242679 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:51:24.242696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.242687 2581 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:51:24.242696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.242695 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:51:24.243098 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.243085 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:51:24.245650 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.245632 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:51:24.252285 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.252248 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:51:24.253177 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.253165 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:51:24.253218 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.253194 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:51:24.253380 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.253367 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:51:24.253428 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.253390 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:51:24.253428 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.253400 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:51:24.253428 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.253407 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:51:24.253428 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.253415 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:51:24.253428 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.253422 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:51:24.254037 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.254028 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:51:24.254064 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.254037 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:51:24.254064 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.254045 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:51:24.254064 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.254057 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:51:24.254140 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.254070 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:51:24.254140 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.254079 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:51:24.254808 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.254799 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:51:24.254808 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.254808 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:51:24.259575 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.259563 2581 server.go:1263] "Started kubelet" Nov 02 19:51:24.259708 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.259676 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:51:24.259825 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.259665 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:51:24.259879 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.259849 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:51:24.260326 ip-10-0-130-111 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:51:24.261252 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.261238 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:51:24.263347 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.263270 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:51:24.269014 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.268994 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 19:51:24.270120 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.270101 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:51:24.270728 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.270711 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:51:24.270885 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.270851 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:24.271565 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271547 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:51:24.271651 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271552 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:51:24.271651 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271585 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:51:24.271651 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271611 2581 factory.go:55] Registering systemd factory Nov 02 19:51:24.271651 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271626 2581 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:51:24.271804 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271703 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:51:24.271804 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271713 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:51:24.271892 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271847 2581 factory.go:153] Registering CRI-O factory Nov 02 19:51:24.271892 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271860 2581 factory.go:223] Registration of the crio container factory successfully Nov 02 19:51:24.271976 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271908 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:51:24.271976 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271926 2581 factory.go:103] Registering Raw factory Nov 02 19:51:24.271976 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.271940 2581 manager.go:1199] Started watching for new ooms in manager Nov 02 19:51:24.273026 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.273010 2581 manager.go:319] Starting recovery of all containers Nov 02 19:51:24.281072 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.281037 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:51:24.287970 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.287959 2581 manager.go:324] Recovery completed Nov 02 19:51:24.290399 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.290382 2581 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 02 19:51:24.293287 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.293259 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:24.295116 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.295098 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:24.295201 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.295135 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:24.295201 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.295152 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:24.295666 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.295650 2581 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:51:24.295666 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.295663 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:51:24.295799 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.295671 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:24.295799 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.295682 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:51:24.296375 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.296359 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:24.296432 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.296396 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:24.296432 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.296412 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:24.297968 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.297952 2581 policy_none.go:50] "Start" Nov 02 19:51:24.298024 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.297974 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:51:24.298024 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.297988 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:51:24.300173 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.300154 2581 policy_none.go:44] "Start" Nov 02 19:51:24.339970 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.339934 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:51:24.343071 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.343054 2581 manager.go:341] "Starting Device Plugin manager" Nov 02 19:51:24.343159 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.343087 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:51:24.343159 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.343097 2581 server.go:87] "Starting device plugin registration server" Nov 02 19:51:24.343337 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.343326 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:51:24.343372 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.343338 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:51:24.343486 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.343457 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:51:24.343660 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.343541 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:51:24.343660 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.343549 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:51:24.344096 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.344064 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:51:24.344168 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.344143 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:24.394609 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.394580 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-111.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 19:51:24.394609 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.394579 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-111.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 19:51:24.395761 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.394593 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-111.us-west-2.compute.internal.18e45256c6fe98d5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-111.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-111.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:51:24.259539157 +0000 UTC m=+0.347878057,LastTimestamp:2026-11-02 19:51:24.259539157 +0000 UTC m=+0.347878057,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-111.us-west-2.compute.internal,}" Nov 02 19:51:24.442475 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.442399 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-111.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 19:51:24.442475 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.442399 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 19:51:24.444559 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.442524 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 19:51:24.444559 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.443722 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:24.444684 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.444661 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:24.444794 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.444695 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:24.444794 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.444706 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:24.444794 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.444740 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.454450 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.454432 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:51:24.454571 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.454455 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:51:24.454571 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.454476 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:51:24.454571 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.454514 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 19:51:24.482915 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.482883 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:51:24.483056 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.482940 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sgjwk" Nov 02 19:51:24.543665 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.543637 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:51:24.549325 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.549302 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.549464 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.549331 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-111.us-west-2.compute.internal\": node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:24.555547 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.555528 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal"] Nov 02 19:51:24.555596 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.555587 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:24.556533 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.556514 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:24.556637 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.556546 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:24.556637 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.556557 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:24.558861 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.558842 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:24.559040 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.559026 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.559078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.559058 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:24.559696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.559679 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:24.559769 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.559709 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:24.559769 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.559721 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:24.559769 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.559726 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:24.559769 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.559761 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:24.559902 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.559776 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:24.561939 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.561923 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.562033 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.561947 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:24.562730 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.562712 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:24.562801 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.562746 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:24.562801 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.562760 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:24.573163 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.573020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.573222 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.573177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.573222 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.573193 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee4b9ca33e3caae4f62b24687213e3f8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"ee4b9ca33e3caae4f62b24687213e3f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.589072 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.589047 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" node="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.592082 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.592064 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" node="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.657401 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.657374 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:24.673617 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.673593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.673688 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.673622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.673688 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.673642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee4b9ca33e3caae4f62b24687213e3f8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"ee4b9ca33e3caae4f62b24687213e3f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.673761 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.673700 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.673761 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.673704 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee4b9ca33e3caae4f62b24687213e3f8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"ee4b9ca33e3caae4f62b24687213e3f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.673761 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.673705 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.757770 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.757716 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:24.858120 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.858091 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:24.892071 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.892046 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.895346 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:24.895324 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:24.958550 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:24.958506 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:25.058794 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:25.058712 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:25.159404 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:25.159368 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:25.197230 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.197210 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 19:51:25.197623 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.197390 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 19:51:25.197623 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:25.197382 2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a83af7d525ec44176aabcd0e4302dda2-d27cf1d047af92e6.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-111.us-west-2.compute.internal.18e45256c91e1c0d\": read tcp 10.0.130.111:56488->13.184.47.174:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-111.us-west-2.compute.internal.18e45256c91e1c0d default 5188 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-111.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-111.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-111.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:51:24 +0000 UTC,LastTimestamp:2026-11-02 19:51:24.556563001 +0000 UTC m=+0.644901902,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-111.us-west-2.compute.internal,}" Nov 02 19:51:25.260528 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:25.260496 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:25.361716 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:25.361649 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:25.455428 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.455406 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:51:25.456331 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.456277 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:51:25.456331 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.456331 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:51:25.456462 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.456344 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:51:25.462319 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:25.462304 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:25.488117 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.488089 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:46:25 +0000 UTC" deadline="2028-07-03 07:28:56.630763909 +0000 UTC" Nov 02 19:51:25.488117 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.488114 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14603h37m31.142653102s" Nov 02 19:51:25.563403 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:25.563369 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:51:25.621995 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.621937 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:51:25.672068 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.672040 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:25.721219 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.721196 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:51:25.736673 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.736654 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:51:25.738155 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.738141 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:51:25.798209 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.798184 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:51:25.813978 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:25.813959 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:51:26.117786 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.117749 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4b9ca33e3caae4f62b24687213e3f8.slice/crio-0786d1a3cabedf3c927e808edce92df0ddb6e74539c11dc41f10b99b669268f6 WatchSource:0}: Error finding container 0786d1a3cabedf3c927e808edce92df0ddb6e74539c11dc41f10b99b669268f6: Status 404 returned error can't find the container with id 0786d1a3cabedf3c927e808edce92df0ddb6e74539c11dc41f10b99b669268f6 Nov 02 19:51:26.122748 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.122728 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:51:26.127623 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.127603 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fa11bb17868be078466e66734ae46b.slice/crio-a8d0b395b02a2125f88670ba84c7550de9ea2f3e14bdb66b6ef37b54da72864d WatchSource:0}: Error finding container a8d0b395b02a2125f88670ba84c7550de9ea2f3e14bdb66b6ef37b54da72864d: Status 404 returned error can't find the container with id a8d0b395b02a2125f88670ba84c7550de9ea2f3e14bdb66b6ef37b54da72864d Nov 02 19:51:26.245426 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.245387 2581 apiserver.go:52] "Watching apiserver" Nov 02 19:51:26.271515 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.271440 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 19:51:26.357629 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.357590 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:51:26.357940 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.357923 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-nc9rw","openshift-multus/multus-additional-cni-plugins-bn5zl","openshift-multus/multus-gspv7","openshift-ovn-kubernetes/ovnkube-node-7m6dl","kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-2nprn","openshift-network-diagnostics/network-check-target-zg4fd","openshift-image-registry/node-ca-rmpzb","kube-system/global-pull-secret-syncer-wkg2t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-7nt8t","openshift-network-operator/iptables-alerter-pvp2h","kube-system/konnectivity-agent-z28kx"] Nov 02 19:51:26.361770 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.361751 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:26.361839 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.361821 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:26.363919 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.363905 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.366102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.366081 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.368452 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.368432 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.370808 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.370787 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.373149 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.373130 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.375917 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.375896 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:26.376019 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.375953 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:26.378321 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.378303 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.381874 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.381849 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.382251 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382228 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.382341 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382264 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-modprobe-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.382341 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-daemon-config\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.382341 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382326 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-etc-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.382486 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382342 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-env-overrides\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.382486 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382407 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-slash\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.382486 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-systemd\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.382486 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-bin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.382666 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-etc-kubernetes\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.382666 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382529 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-conf-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.382666 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-multus-certs\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.382666 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382619 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-config\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.382666 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382653 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7j2\" (UniqueName: \"kubernetes.io/projected/f81e0175-b8d3-4348-bc49-d9e00531643f-kube-api-access-hk7j2\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.382878 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382708 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-multus\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.382878 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-kubelet\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.382878 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382761 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b39d4538-dc20-4f6e-8170-4f5c0327cf69-serviceca\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.382878 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-system-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.382878 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-systemd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.382878 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382862 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtpvg\" (UniqueName: \"kubernetes.io/projected/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-kube-api-access-xtpvg\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-var-lib-kubelet\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382900 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-log-socket\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-etc-tuned\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.382975 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-kubelet\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383005 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383032 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383065 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383105 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cnibin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383137 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cni-binary-copy\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.383162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-k8s-cni-cncf-io\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383186 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plp8r\" (UniqueName: \"kubernetes.io/projected/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-kube-api-access-plp8r\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383203 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-bin\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383218 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-netd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f81e0175-b8d3-4348-bc49-d9e00531643f-ovn-node-metrics-cert\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383250 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-systemd-units\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383274 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-node-log\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383314 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-tmp-dir\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383335 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-kubernetes\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383352 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-conf\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-run\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383420 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-sys\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383459 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-lib-modules\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383493 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-socket-dir-parent\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383518 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-netns\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383541 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-ovn\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383563 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sng9v\" (UniqueName: \"kubernetes.io/projected/41e23004-aede-47df-a301-b880f169a803-kube-api-access-sng9v\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.383690 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-hostroot\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.384262 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383626 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-netns\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.384262 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383658 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-hosts-file\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.384262 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-tmp\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.384262 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-script-lib\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.384262 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383747 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysconfig\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.384262 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383771 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:26.384262 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383818 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-host\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.384262 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383845 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39d4538-dc20-4f6e-8170-4f5c0327cf69-host\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.384262 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383902 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcg4w\" (UniqueName: \"kubernetes.io/projected/b39d4538-dc20-4f6e-8170-4f5c0327cf69-kube-api-access-rcg4w\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.384262 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.383939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-var-lib-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.384788 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.384389 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.386928 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.386911 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:26.389740 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.389716 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:26.389838 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.389781 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:26.420869 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.420849 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:51:26.420869 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.420865 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:51:26.421037 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.420887 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:51:26.421037 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.420901 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vl78h\"" Nov 02 19:51:26.422191 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.422175 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:51:26.422424 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.422410 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:51:26.422514 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.422442 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:51:26.422514 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.422445 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:51:26.422514 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.422467 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-shcvr\"" Nov 02 19:51:26.424764 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.424716 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:51:26.424764 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.424757 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:51:26.424894 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.424787 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlz48\"" Nov 02 19:51:26.424894 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.424844 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:51:26.424989 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.424898 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:51:26.424989 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.424939 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:51:26.425051 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.425016 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:51:26.426625 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.426603 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:51:26.458356 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.458302 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" event={"ID":"27fa11bb17868be078466e66734ae46b","Type":"ContainerStarted","Data":"a8d0b395b02a2125f88670ba84c7550de9ea2f3e14bdb66b6ef37b54da72864d"} Nov 02 19:51:26.459251 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.459233 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" event={"ID":"ee4b9ca33e3caae4f62b24687213e3f8","Type":"ContainerStarted","Data":"0786d1a3cabedf3c927e808edce92df0ddb6e74539c11dc41f10b99b669268f6"} Nov 02 19:51:26.468810 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.468786 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:51:26.468810 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.468787 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:51:26.472260 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.472243 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:51:26.472583 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.472566 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:51:26.472637 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.472571 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:51:26.475365 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.475352 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dbl8w\"" Nov 02 19:51:26.475427 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.475415 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4w4l2\"" Nov 02 19:51:26.476704 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.476690 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:51:26.476769 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.476731 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:51:26.476845 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.476800 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qqgmv\"" Nov 02 19:51:26.476933 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.476920 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:51:26.476989 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.476922 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:51:26.476989 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.476985 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:51:26.479248 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.479231 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4hfb\"" Nov 02 19:51:26.479248 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.479234 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8m7zt\"" Nov 02 19:51:26.479450 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.479418 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:51:26.479450 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.479433 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:51:26.479450 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.479439 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:51:26.479580 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.479444 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:51:26.479580 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.479476 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:51:26.479580 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.479524 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6fsk8" Nov 02 19:51:26.479580 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.479530 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f8f9p\"" Nov 02 19:51:26.484465 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-var-lib-kubelet\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.484558 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484469 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.484558 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484486 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-sys-fs\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.484558 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-log-socket\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.484558 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484550 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-var-lib-kubelet\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.484696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484553 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.484696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484575 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-log-socket\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.484696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484584 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-etc-tuned\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.484696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-kubelet-config\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:26.484696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484628 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-iptables-alerter-script\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.484696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484647 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-kubelet\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.484696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484661 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.484696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.484696 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484683 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484723 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484759 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484784 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-kubelet\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484807 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cnibin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484855 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cni-binary-copy\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484858 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484903 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cnibin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484885 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-k8s-cni-cncf-io\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484946 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plp8r\" (UniqueName: \"kubernetes.io/projected/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-kube-api-access-plp8r\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484933 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-k8s-cni-cncf-io\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.484985 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-bin\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485010 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-netd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485031 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f81e0175-b8d3-4348-bc49-d9e00531643f-ovn-node-metrics-cert\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485039 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-bin\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485054 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-systemd-units\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485102 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485075 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-systemd-units\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485083 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-node-log\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485098 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-node-log\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-tmp-dir\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-kubernetes\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-conf\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485186 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-run\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485201 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-kubernetes\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485209 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-sys\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485231 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-netd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-lib-modules\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485254 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-sys\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485277 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-run\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485321 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-socket-dir-parent\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485333 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-conf\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485341 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-netns\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485361 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-lib-modules\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485376 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-socket-dir-parent\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.485966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485357 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-netns\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485385 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-ovn\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485407 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-ovn\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485416 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sng9v\" (UniqueName: \"kubernetes.io/projected/41e23004-aede-47df-a301-b880f169a803-kube-api-access-sng9v\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485442 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-registration-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485463 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-hostroot\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-netns\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-hosts-file\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485523 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-tmp\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485542 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-hostroot\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485549 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-netns\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485559 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6hm\" (UniqueName: \"kubernetes.io/projected/ae8ba755-ec98-496c-a59a-4f4625fea774-kube-api-access-5m6hm\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485653 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-hosts-file\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485688 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkn9t\" (UniqueName: \"kubernetes.io/projected/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-kube-api-access-jkn9t\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485708 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cnibin\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.485951 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-tmp-dir\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.486695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486012 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cni-binary-copy\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486250 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-script-lib\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysconfig\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486423 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486447 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-system-cni-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486494 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysconfig\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486538 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-host\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486499 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-host\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486583 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486609 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486639 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39d4538-dc20-4f6e-8170-4f5c0327cf69-host\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486666 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcg4w\" (UniqueName: \"kubernetes.io/projected/b39d4538-dc20-4f6e-8170-4f5c0327cf69-kube-api-access-rcg4w\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-var-lib-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486730 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39d4538-dc20-4f6e-8170-4f5c0327cf69-host\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-modprobe-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.487517 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486777 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-daemon-config\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486804 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-etc-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-env-overrides\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486825 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-script-lib\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486864 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9z9n\" (UniqueName: \"kubernetes.io/projected/cd701aaf-13ea-4699-9c01-5167947a614a-kube-api-access-w9z9n\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486900 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486938 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-etc-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.486805 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-modprobe-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487104 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-var-lib-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487131 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487168 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-slash\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487239 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-systemd\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487285 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-env-overrides\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c6469de-dc6b-405f-b14f-cf765fd9d06f-agent-certs\") pod \"konnectivity-agent-z28kx\" (UID: \"4c6469de-dc6b-405f-b14f-cf765fd9d06f\") " pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487352 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-systemd\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.488310 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487415 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-host-slash\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487438 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-daemon-config\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-bin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487444 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-slash\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487473 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-bin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487476 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-etc-kubernetes\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487523 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-etc-kubernetes\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487560 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-socket-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487625 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-device-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-conf-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487714 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-conf-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-multus-certs\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-config\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487817 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7j2\" (UniqueName: \"kubernetes.io/projected/f81e0175-b8d3-4348-bc49-d9e00531643f-kube-api-access-hk7j2\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487862 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c6469de-dc6b-405f-b14f-cf765fd9d06f-konnectivity-ca\") pod \"konnectivity-agent-z28kx\" (UID: \"4c6469de-dc6b-405f-b14f-cf765fd9d06f\") " pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487861 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-multus-certs\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487906 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-multus\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487935 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-multus\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.488973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.487969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-kubelet\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488001 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-dbus\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488038 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-kubelet\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488045 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9jvj\" (UniqueName: \"kubernetes.io/projected/6048b8aa-9e93-4214-8b5a-ed6663b632b5-kube-api-access-l9jvj\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488087 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b39d4538-dc20-4f6e-8170-4f5c0327cf69-serviceca\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-system-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-systemd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488236 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-system-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488283 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-config\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488290 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtpvg\" (UniqueName: \"kubernetes.io/projected/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-kube-api-access-xtpvg\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488374 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-systemd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488523 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b39d4538-dc20-4f6e-8170-4f5c0327cf69-serviceca\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488679 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-etc-tuned\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488748 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-tmp\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.489796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.488825 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f81e0175-b8d3-4348-bc49-d9e00531643f-ovn-node-metrics-cert\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.501970 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.501947 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:51:26.545114 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.545057 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:26.545114 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.545078 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:26.545114 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.545091 2581 projected.go:196] Error preparing data for projected volume kube-api-access-psrcq for pod openshift-network-diagnostics/network-check-target-zg4fd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:26.545273 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.545168 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq podName:040d6f5e-129c-4665-ad54-957415093fa3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:27.045139708 +0000 UTC m=+3.133478611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-psrcq" (UniqueName: "kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq") pod "network-check-target-zg4fd" (UID: "040d6f5e-129c-4665-ad54-957415093fa3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:26.546555 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.546530 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plp8r\" (UniqueName: \"kubernetes.io/projected/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-kube-api-access-plp8r\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.546645 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.546535 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sng9v\" (UniqueName: \"kubernetes.io/projected/41e23004-aede-47df-a301-b880f169a803-kube-api-access-sng9v\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.547781 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.547752 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcg4w\" (UniqueName: \"kubernetes.io/projected/b39d4538-dc20-4f6e-8170-4f5c0327cf69-kube-api-access-rcg4w\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.548336 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.548305 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7j2\" (UniqueName: \"kubernetes.io/projected/f81e0175-b8d3-4348-bc49-d9e00531643f-kube-api-access-hk7j2\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.548501 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.548484 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtpvg\" (UniqueName: \"kubernetes.io/projected/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-kube-api-access-xtpvg\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.588601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588573 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:26.588601 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-system-cni-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.588797 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588632 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.588797 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588649 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.588797 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588706 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-system-cni-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.588797 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9z9n\" (UniqueName: \"kubernetes.io/projected/cd701aaf-13ea-4699-9c01-5167947a614a-kube-api-access-w9z9n\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:26.588797 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.588747 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:26.588797 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588766 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.588797 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588814 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c6469de-dc6b-405f-b14f-cf765fd9d06f-agent-certs\") pod \"konnectivity-agent-z28kx\" (UID: \"4c6469de-dc6b-405f-b14f-cf765fd9d06f\") " pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.588855 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret podName:76aa7437-fa4a-42a6-90d5-9043d9b8a71d nodeName:}" failed. No retries permitted until 2026-11-02 19:51:27.088822118 +0000 UTC m=+3.177161015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret") pod "global-pull-secret-syncer-wkg2t" (UID: "76aa7437-fa4a-42a6-90d5-9043d9b8a71d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588900 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-host-slash\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588949 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-socket-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588963 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.588977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-device-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c6469de-dc6b-405f-b14f-cf765fd9d06f-konnectivity-ca\") pod \"konnectivity-agent-z28kx\" (UID: \"4c6469de-dc6b-405f-b14f-cf765fd9d06f\") " pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.589027 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589037 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-device-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:26.589086 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs podName:cd701aaf-13ea-4699-9c01-5167947a614a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:27.089070518 +0000 UTC m=+3.177409421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs") pod "network-metrics-daemon-7nt8t" (UID: "cd701aaf-13ea-4699-9c01-5167947a614a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589085 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-dbus\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589092 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-host-slash\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.589099 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-dbus\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jvj\" (UniqueName: \"kubernetes.io/projected/6048b8aa-9e93-4214-8b5a-ed6663b632b5-kube-api-access-l9jvj\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589186 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-sys-fs\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-kubelet-config\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589218 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-iptables-alerter-script\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589245 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589248 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-registration-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589274 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589290 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6hm\" (UniqueName: \"kubernetes.io/projected/ae8ba755-ec98-496c-a59a-4f4625fea774-kube-api-access-5m6hm\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589319 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkn9t\" (UniqueName: \"kubernetes.io/projected/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-kube-api-access-jkn9t\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cnibin\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589390 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cnibin\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589431 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-sys-fs\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589480 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-kubelet-config\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:26.589552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589504 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.590273 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589616 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-socket-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.590273 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589677 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-registration-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.590273 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.589883 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.590273 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.590022 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.590273 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.590091 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c6469de-dc6b-405f-b14f-cf765fd9d06f-konnectivity-ca\") pod \"konnectivity-agent-z28kx\" (UID: \"4c6469de-dc6b-405f-b14f-cf765fd9d06f\") " pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:26.590273 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.590098 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-iptables-alerter-script\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.590273 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.590222 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.591636 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.591619 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c6469de-dc6b-405f-b14f-cf765fd9d06f-agent-certs\") pod \"konnectivity-agent-z28kx\" (UID: \"4c6469de-dc6b-405f-b14f-cf765fd9d06f\") " pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:26.658743 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.658706 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jvj\" (UniqueName: \"kubernetes.io/projected/6048b8aa-9e93-4214-8b5a-ed6663b632b5-kube-api-access-l9jvj\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.658892 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.658713 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkn9t\" (UniqueName: \"kubernetes.io/projected/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-kube-api-access-jkn9t\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.658892 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.658764 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6hm\" (UniqueName: \"kubernetes.io/projected/ae8ba755-ec98-496c-a59a-4f4625fea774-kube-api-access-5m6hm\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.659035 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.659017 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9z9n\" (UniqueName: \"kubernetes.io/projected/cd701aaf-13ea-4699-9c01-5167947a614a-kube-api-access-w9z9n\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:26.675996 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.675979 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:51:26.682580 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.682565 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gspv7" Nov 02 19:51:26.686306 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.686270 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39d4538_dc20_4f6e_8170_4f5c0327cf69.slice/crio-31155f41c7c63eab90ba61785de776eb7e706838aad039e6c0f93846777f08b9 WatchSource:0}: Error finding container 31155f41c7c63eab90ba61785de776eb7e706838aad039e6c0f93846777f08b9: Status 404 returned error can't find the container with id 31155f41c7c63eab90ba61785de776eb7e706838aad039e6c0f93846777f08b9 Nov 02 19:51:26.691407 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.691384 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd8e8aa_3e6c_40de_bce8_3b6d3fc194d2.slice/crio-7f6564f9212779fbfcdac37d7ba506c3720115a643f6c0f7cbf51749d28b6e5e WatchSource:0}: Error finding container 7f6564f9212779fbfcdac37d7ba506c3720115a643f6c0f7cbf51749d28b6e5e: Status 404 returned error can't find the container with id 7f6564f9212779fbfcdac37d7ba506c3720115a643f6c0f7cbf51749d28b6e5e Nov 02 19:51:26.701729 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.701708 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:51:26.702816 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.702800 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:51:26.709072 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.709057 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:26.711242 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.711222 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:51:26.711668 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.711644 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e35ec0a_ad15_48bc_ac59_e08c8dec75a0.slice/crio-f2be4ef9fd6b4e93ddbb046eed8ef46af779711960a8902397a7dcda71309f9f WatchSource:0}: Error finding container f2be4ef9fd6b4e93ddbb046eed8ef46af779711960a8902397a7dcda71309f9f: Status 404 returned error can't find the container with id f2be4ef9fd6b4e93ddbb046eed8ef46af779711960a8902397a7dcda71309f9f Nov 02 19:51:26.713653 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.713633 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e23004_aede_47df_a301_b880f169a803.slice/crio-df7524b840bddc67bb917fb7a4d3f3669e5fa29d42ddeff5e8012697bea50971 WatchSource:0}: Error finding container df7524b840bddc67bb917fb7a4d3f3669e5fa29d42ddeff5e8012697bea50971: Status 404 returned error can't find the container with id df7524b840bddc67bb917fb7a4d3f3669e5fa29d42ddeff5e8012697bea50971 Nov 02 19:51:26.720106 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.720085 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81e0175_b8d3_4348_bc49_d9e00531643f.slice/crio-2ff900718fa6f214f64b5aadb7e4ece46156eb00d770996e5aa287a04ffc9ba1 WatchSource:0}: Error finding container 2ff900718fa6f214f64b5aadb7e4ece46156eb00d770996e5aa287a04ffc9ba1: Status 404 returned error can't find the container with id 2ff900718fa6f214f64b5aadb7e4ece46156eb00d770996e5aa287a04ffc9ba1 Nov 02 19:51:26.723115 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.723092 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e71e3e7_20f2_4dbd_bd77_1debbbdacf68.slice/crio-04212e999252863e939f0a718697e75636fa07bc8c39f1b4c2c9181da3afcd97 WatchSource:0}: Error finding container 04212e999252863e939f0a718697e75636fa07bc8c39f1b4c2c9181da3afcd97: Status 404 returned error can't find the container with id 04212e999252863e939f0a718697e75636fa07bc8c39f1b4c2c9181da3afcd97 Nov 02 19:51:26.730752 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.730738 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:26.732899 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.732879 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:51:26.734437 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:26.734419 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:51:26.741744 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.741725 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6469de_dc6b_405f_b14f_cf765fd9d06f.slice/crio-a7a02aa8ada6156560906284b23afa0a0b77373d07eeb8b1146adf633865b31f WatchSource:0}: Error finding container a7a02aa8ada6156560906284b23afa0a0b77373d07eeb8b1146adf633865b31f: Status 404 returned error can't find the container with id a7a02aa8ada6156560906284b23afa0a0b77373d07eeb8b1146adf633865b31f Nov 02 19:51:26.744652 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.744630 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8ba755_ec98_496c_a59a_4f4625fea774.slice/crio-0c4cb6289638aa3a12a91a97de8d4b333376eb68b0eb0acbcd8dcfb9f15cdd3b WatchSource:0}: Error finding container 0c4cb6289638aa3a12a91a97de8d4b333376eb68b0eb0acbcd8dcfb9f15cdd3b: Status 404 returned error can't find the container with id 0c4cb6289638aa3a12a91a97de8d4b333376eb68b0eb0acbcd8dcfb9f15cdd3b Nov 02 19:51:26.747926 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:51:26.747907 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6048b8aa_9e93_4214_8b5a_ed6663b632b5.slice/crio-d358bf5747cf075a43e90306382ce0c282d51d4d5aefd7c1ccc42b7256f5e3c1 WatchSource:0}: Error finding container d358bf5747cf075a43e90306382ce0c282d51d4d5aefd7c1ccc42b7256f5e3c1: Status 404 returned error can't find the container with id d358bf5747cf075a43e90306382ce0c282d51d4d5aefd7c1ccc42b7256f5e3c1 Nov 02 19:51:27.093142 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.093100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:27.093142 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.093140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:27.093403 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.093169 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:27.093403 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:27.093277 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:27.093403 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:27.093299 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:27.093403 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:27.093326 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:27.093403 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:27.093340 2581 projected.go:196] Error preparing data for projected volume kube-api-access-psrcq for pod openshift-network-diagnostics/network-check-target-zg4fd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:27.093403 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:27.093351 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs podName:cd701aaf-13ea-4699-9c01-5167947a614a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:28.093331558 +0000 UTC m=+4.181670447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs") pod "network-metrics-daemon-7nt8t" (UID: "cd701aaf-13ea-4699-9c01-5167947a614a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:27.093403 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:27.093305 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:27.093403 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:27.093397 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq podName:040d6f5e-129c-4665-ad54-957415093fa3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:28.0933799 +0000 UTC m=+4.181718796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-psrcq" (UniqueName: "kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq") pod "network-check-target-zg4fd" (UID: "040d6f5e-129c-4665-ad54-957415093fa3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:27.093827 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:27.093429 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret podName:76aa7437-fa4a-42a6-90d5-9043d9b8a71d nodeName:}" failed. No retries permitted until 2026-11-02 19:51:28.09341705 +0000 UTC m=+4.181755938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret") pod "global-pull-secret-syncer-wkg2t" (UID: "76aa7437-fa4a-42a6-90d5-9043d9b8a71d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:27.462528 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.462438 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rmpzb" event={"ID":"b39d4538-dc20-4f6e-8170-4f5c0327cf69","Type":"ContainerStarted","Data":"31155f41c7c63eab90ba61785de776eb7e706838aad039e6c0f93846777f08b9"} Nov 02 19:51:27.463953 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.463924 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" event={"ID":"ae8ba755-ec98-496c-a59a-4f4625fea774","Type":"ContainerStarted","Data":"0c4cb6289638aa3a12a91a97de8d4b333376eb68b0eb0acbcd8dcfb9f15cdd3b"} Nov 02 19:51:27.465424 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.465397 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z28kx" event={"ID":"4c6469de-dc6b-405f-b14f-cf765fd9d06f","Type":"ContainerStarted","Data":"a7a02aa8ada6156560906284b23afa0a0b77373d07eeb8b1146adf633865b31f"} Nov 02 19:51:27.467008 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.466983 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pvp2h" event={"ID":"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68","Type":"ContainerStarted","Data":"04212e999252863e939f0a718697e75636fa07bc8c39f1b4c2c9181da3afcd97"} Nov 02 19:51:27.468718 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.468689 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" event={"ID":"f81e0175-b8d3-4348-bc49-d9e00531643f","Type":"ContainerStarted","Data":"2ff900718fa6f214f64b5aadb7e4ece46156eb00d770996e5aa287a04ffc9ba1"} Nov 02 19:51:27.469902 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.469867 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gspv7" event={"ID":"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2","Type":"ContainerStarted","Data":"7f6564f9212779fbfcdac37d7ba506c3720115a643f6c0f7cbf51749d28b6e5e"} Nov 02 19:51:27.471321 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.471294 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" event={"ID":"6048b8aa-9e93-4214-8b5a-ed6663b632b5","Type":"ContainerStarted","Data":"d358bf5747cf075a43e90306382ce0c282d51d4d5aefd7c1ccc42b7256f5e3c1"} Nov 02 19:51:27.472483 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.472458 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2nprn" event={"ID":"41e23004-aede-47df-a301-b880f169a803","Type":"ContainerStarted","Data":"df7524b840bddc67bb917fb7a4d3f3669e5fa29d42ddeff5e8012697bea50971"} Nov 02 19:51:27.473467 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.473446 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nc9rw" event={"ID":"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0","Type":"ContainerStarted","Data":"f2be4ef9fd6b4e93ddbb046eed8ef46af779711960a8902397a7dcda71309f9f"} Nov 02 19:51:27.482994 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.482965 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:46:27 +0000 UTC" deadline="2028-07-11 12:26:03.502570578 +0000 UTC" Nov 02 19:51:27.482994 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:27.482994 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14800h34m36.01958051s" Nov 02 19:51:28.106981 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:28.106937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:28.107226 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:28.106997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:28.107226 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:28.107038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:28.107226 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.107213 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:28.107418 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.107278 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs podName:cd701aaf-13ea-4699-9c01-5167947a614a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:30.107258341 +0000 UTC m=+6.195597233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs") pod "network-metrics-daemon-7nt8t" (UID: "cd701aaf-13ea-4699-9c01-5167947a614a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:28.107418 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.107369 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:28.107418 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.107409 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret podName:76aa7437-fa4a-42a6-90d5-9043d9b8a71d nodeName:}" failed. No retries permitted until 2026-11-02 19:51:30.107397024 +0000 UTC m=+6.195735912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret") pod "global-pull-secret-syncer-wkg2t" (UID: "76aa7437-fa4a-42a6-90d5-9043d9b8a71d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:28.107583 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.107493 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:28.107583 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.107508 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:28.107583 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.107520 2581 projected.go:196] Error preparing data for projected volume kube-api-access-psrcq for pod openshift-network-diagnostics/network-check-target-zg4fd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:28.107583 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.107558 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq podName:040d6f5e-129c-4665-ad54-957415093fa3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:30.107546328 +0000 UTC m=+6.195885221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-psrcq" (UniqueName: "kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq") pod "network-check-target-zg4fd" (UID: "040d6f5e-129c-4665-ad54-957415093fa3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:28.461809 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:28.461734 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:28.461963 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.461875 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:28.462038 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:28.461966 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:28.462098 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.462046 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:28.462159 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:28.462108 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:28.462213 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:28.462191 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:28.483218 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:28.483155 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:46:27 +0000 UTC" deadline="2028-06-07 14:07:13.56359425 +0000 UTC" Nov 02 19:51:28.483218 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:28.483216 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13986h15m45.080383162s" Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:30.124668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:30.124725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:30.124767 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.124931 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.124996 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs podName:cd701aaf-13ea-4699-9c01-5167947a614a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:34.124978528 +0000 UTC m=+10.213317435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs") pod "network-metrics-daemon-7nt8t" (UID: "cd701aaf-13ea-4699-9c01-5167947a614a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.125070 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.125109 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret podName:76aa7437-fa4a-42a6-90d5-9043d9b8a71d nodeName:}" failed. No retries permitted until 2026-11-02 19:51:34.125096912 +0000 UTC m=+10.213435814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret") pod "global-pull-secret-syncer-wkg2t" (UID: "76aa7437-fa4a-42a6-90d5-9043d9b8a71d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.125188 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.125203 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.125219 2581 projected.go:196] Error preparing data for projected volume kube-api-access-psrcq for pod openshift-network-diagnostics/network-check-target-zg4fd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:30.125289 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.125253 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq podName:040d6f5e-129c-4665-ad54-957415093fa3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:34.125242282 +0000 UTC m=+10.213581188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-psrcq" (UniqueName: "kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq") pod "network-check-target-zg4fd" (UID: "040d6f5e-129c-4665-ad54-957415093fa3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:30.457248 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:30.455420 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:30.457248 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.455556 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:30.457248 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:30.455429 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:30.457248 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:30.455612 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:30.457248 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.455706 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:30.457248 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:30.455809 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:31.502481 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:31.502434 2581 generic.go:356] "Generic (PLEG): container finished" podID="27fa11bb17868be078466e66734ae46b" containerID="5af2354cc68bb91686fd7910e17fabb892d3b2762b44e469435ff9353c28bfe7" exitCode=0 Nov 02 19:51:31.503003 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:31.502526 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" event={"ID":"27fa11bb17868be078466e66734ae46b","Type":"ContainerDied","Data":"5af2354cc68bb91686fd7910e17fabb892d3b2762b44e469435ff9353c28bfe7"} Nov 02 19:51:31.507793 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:31.507762 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" event={"ID":"ee4b9ca33e3caae4f62b24687213e3f8","Type":"ContainerStarted","Data":"d8cebce6667ad01c532a203d5a566b3be2a0a3abcda97df1a5c340c839eaf7ce"} Nov 02 19:51:31.527552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:31.524975 2581 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="22050b034c7f573d08559a6d882fcd268715721e89af3c3b14fe281e5eb2f20f" exitCode=0 Nov 02 19:51:31.527552 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:31.525026 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" event={"ID":"6048b8aa-9e93-4214-8b5a-ed6663b632b5","Type":"ContainerDied","Data":"22050b034c7f573d08559a6d882fcd268715721e89af3c3b14fe281e5eb2f20f"} Nov 02 19:51:31.875061 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:31.872265 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" podStartSLOduration=5.872244784 podStartE2EDuration="5.872244784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:31.746580955 +0000 UTC m=+7.834919866" watchObservedRunningTime="2026-11-02 19:51:31.872244784 +0000 UTC m=+7.960583697" Nov 02 19:51:32.457956 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:32.457442 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:32.457956 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:32.457597 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:32.457956 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:32.457693 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:32.457956 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:32.457787 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:32.457956 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:32.457854 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:32.457956 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:32.457931 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:32.532367 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:32.531598 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" event={"ID":"27fa11bb17868be078466e66734ae46b","Type":"ContainerStarted","Data":"64e8448abf20e7ae50711c531a4fe87a02a4ff0042f9d972613b91e909b45644"} Nov 02 19:51:32.536463 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:32.536375 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z28kx" event={"ID":"4c6469de-dc6b-405f-b14f-cf765fd9d06f","Type":"ContainerStarted","Data":"c7eb0034120a2b13d085691cb5792f5cd436821423bf5cb1b9818047e60868e7"} Nov 02 19:51:32.782597 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:32.782263 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" podStartSLOduration=6.7822404160000005 podStartE2EDuration="6.782240416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:32.658630816 +0000 UTC m=+8.746969729" watchObservedRunningTime="2026-11-02 19:51:32.782240416 +0000 UTC m=+8.870579330" Nov 02 19:51:32.782597 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:32.782490 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z28kx" podStartSLOduration=2.601873506 podStartE2EDuration="7.782481965s" totalImagesPullingTime="5.180608459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:32.781595735 +0000 UTC m=+8.869934646" watchObservedRunningTime="2026-11-02 19:51:32.782481965 +0000 UTC m=+8.870820876" Nov 02 19:51:33.535455 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:33.535170 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:33.536694 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:33.536415 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:33.544620 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:33.543500 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rmpzb" event={"ID":"b39d4538-dc20-4f6e-8170-4f5c0327cf69","Type":"ContainerStarted","Data":"27a669150e7cd0c98111dc8e4c238973d53d6272ba6ad4b52a597ec37d024bb4"} Nov 02 19:51:33.544620 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:33.544579 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:33.545477 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:33.545394 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:51:33.918192 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:33.917729 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rmpzb" podStartSLOduration=2.849663467 podStartE2EDuration="8.917710234s" totalImagesPullingTime="6.068046767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:33.917575054 +0000 UTC m=+10.005913967" watchObservedRunningTime="2026-11-02 19:51:33.917710234 +0000 UTC m=+10.006049148" Nov 02 19:51:34.161122 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:34.161033 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:34.161305 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:34.161136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:34.161305 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:34.161183 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:34.161444 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.161366 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:34.161444 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.161387 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:34.161444 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.161402 2581 projected.go:196] Error preparing data for projected volume kube-api-access-psrcq for pod openshift-network-diagnostics/network-check-target-zg4fd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:34.161600 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.161465 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq podName:040d6f5e-129c-4665-ad54-957415093fa3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:42.161445352 +0000 UTC m=+18.249784266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-psrcq" (UniqueName: "kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq") pod "network-check-target-zg4fd" (UID: "040d6f5e-129c-4665-ad54-957415093fa3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:34.161600 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.161552 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:34.161600 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.161596 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs podName:cd701aaf-13ea-4699-9c01-5167947a614a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:42.161582585 +0000 UTC m=+18.249921488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs") pod "network-metrics-daemon-7nt8t" (UID: "cd701aaf-13ea-4699-9c01-5167947a614a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:34.161771 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.161657 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:34.161771 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.161694 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret podName:76aa7437-fa4a-42a6-90d5-9043d9b8a71d nodeName:}" failed. No retries permitted until 2026-11-02 19:51:42.161682503 +0000 UTC m=+18.250021406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret") pod "global-pull-secret-syncer-wkg2t" (UID: "76aa7437-fa4a-42a6-90d5-9043d9b8a71d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:34.458578 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:34.455894 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:34.458578 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.456046 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:34.458578 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:34.456126 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:34.458578 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.456198 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:34.458578 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:34.456255 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:34.458578 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:34.456349 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:34.551350 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:34.551313 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" event={"ID":"ae8ba755-ec98-496c-a59a-4f4625fea774","Type":"ContainerStarted","Data":"67add24aef1c9eb1f5c3d36e01f6ddb51741918cf21b9639c3caf9f6cc7c9632"} Nov 02 19:51:36.139186 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.139156 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:51:36.355491 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.355263 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:51:36.13917884Z","UUID":"01576b04-d440-4f71-b484-ddd2a09f4d7e","Handler":null,"Name":"","Endpoint":""} Nov 02 19:51:36.356990 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.356958 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:51:36.356990 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.356991 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:51:36.460120 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.460098 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:36.460240 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:36.460213 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:36.460328 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.460305 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:36.460419 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:36.460400 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:36.460480 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.460468 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:36.460563 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:36.460540 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:36.556512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.556470 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nc9rw" event={"ID":"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0","Type":"ContainerStarted","Data":"ddba04cdb3f7dbc686f97ca7a9a641c7b851c953d0d750c12eca2bd05af48b89"} Nov 02 19:51:36.559021 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.558991 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" event={"ID":"ae8ba755-ec98-496c-a59a-4f4625fea774","Type":"ContainerStarted","Data":"b8f951cf5d779dea7ecd12d94127080bd593703feee5447f6724e5967028a3ea"} Nov 02 19:51:36.560630 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.560596 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gspv7" event={"ID":"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2","Type":"ContainerStarted","Data":"5904757d10f2f9cc4fdbe6f8ab3d7ce171af6fec819f3a73f3e40105eb1aca84"} Nov 02 19:51:36.563114 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.563088 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" event={"ID":"6048b8aa-9e93-4214-8b5a-ed6663b632b5","Type":"ContainerStarted","Data":"dcd309aa3b6e63d71e8db375f5b178296e14a3796e3500f7b070afd49bdca524"} Nov 02 19:51:36.785366 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.785138 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nc9rw" podStartSLOduration=2.842243296 podStartE2EDuration="11.785124157s" totalImagesPullingTime="8.942880861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:36.668509848 +0000 UTC m=+12.756848758" watchObservedRunningTime="2026-11-02 19:51:36.785124157 +0000 UTC m=+12.873463064" Nov 02 19:51:36.785533 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:36.785484 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gspv7" podStartSLOduration=2.960639317 podStartE2EDuration="11.785478086s" totalImagesPullingTime="8.824838769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:36.785024619 +0000 UTC m=+12.873363529" watchObservedRunningTime="2026-11-02 19:51:36.785478086 +0000 UTC m=+12.873816994" Nov 02 19:51:37.567997 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:37.567961 2581 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="dcd309aa3b6e63d71e8db375f5b178296e14a3796e3500f7b070afd49bdca524" exitCode=0 Nov 02 19:51:37.568476 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:37.568064 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" event={"ID":"6048b8aa-9e93-4214-8b5a-ed6663b632b5","Type":"ContainerDied","Data":"dcd309aa3b6e63d71e8db375f5b178296e14a3796e3500f7b070afd49bdca524"} Nov 02 19:51:37.571242 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:37.571218 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pvp2h" event={"ID":"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68","Type":"ContainerStarted","Data":"9031f61194b7ca292591ccb0806a86722f768d4e099eeb2edad6ae16ecd6a295"} Nov 02 19:51:38.455411 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:38.455380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:38.455557 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:38.455487 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:38.455557 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:38.455517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:38.455623 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:38.455585 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:38.455623 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:38.455618 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:38.455678 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:38.455656 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:38.575350 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:38.575072 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2nprn" event={"ID":"41e23004-aede-47df-a301-b880f169a803","Type":"ContainerStarted","Data":"f2f78ab74ae77147fdc4f346672c082f7ac5f5a64ab1b64f6bc8dd3533fc9964"} Nov 02 19:51:38.578229 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:38.578196 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" event={"ID":"ae8ba755-ec98-496c-a59a-4f4625fea774","Type":"ContainerStarted","Data":"edc15652954ff25a12fef6dc52bef70918d455b3075bdf5cff0b0dd0e3395b48"} Nov 02 19:51:38.688859 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:38.688798 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pvp2h" podStartSLOduration=4.472455012 podStartE2EDuration="13.688781487s" totalImagesPullingTime="9.216326475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:37.88869603 +0000 UTC m=+13.977034939" watchObservedRunningTime="2026-11-02 19:51:38.688781487 +0000 UTC m=+14.777120396" Nov 02 19:51:38.689184 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:38.689160 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2nprn" podStartSLOduration=2.097557884 podStartE2EDuration="13.689152336s" totalImagesPullingTime="11.591594452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:38.688666473 +0000 UTC m=+14.777005382" watchObservedRunningTime="2026-11-02 19:51:38.689152336 +0000 UTC m=+14.777491243" Nov 02 19:51:38.802454 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:38.802403 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" podStartSLOduration=2.663818892 podStartE2EDuration="13.80238622s" totalImagesPullingTime="11.138567328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:38.801845928 +0000 UTC m=+14.890184835" watchObservedRunningTime="2026-11-02 19:51:38.80238622 +0000 UTC m=+14.890725129" Nov 02 19:51:39.583193 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:39.583133 2581 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="bdfb4b1e2dd96a5d8576f43a6ab0dcd19f39e01aa87398e7b07e3e0b0fd9d03c" exitCode=0 Nov 02 19:51:39.583864 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:39.583215 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" event={"ID":"6048b8aa-9e93-4214-8b5a-ed6663b632b5","Type":"ContainerDied","Data":"bdfb4b1e2dd96a5d8576f43a6ab0dcd19f39e01aa87398e7b07e3e0b0fd9d03c"} Nov 02 19:51:40.454813 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:40.454788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:40.454957 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:40.454787 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:40.454957 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:40.454884 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:40.455055 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:40.454960 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:40.455055 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:40.454795 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:40.455055 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:40.455039 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:41.590635 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:41.590415 2581 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="4a05283823901fb0d6df87c0d2acf3972915d81fc91f8f514bf8bdab3857f0ca" exitCode=0 Nov 02 19:51:41.590635 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:41.590498 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" event={"ID":"6048b8aa-9e93-4214-8b5a-ed6663b632b5","Type":"ContainerDied","Data":"4a05283823901fb0d6df87c0d2acf3972915d81fc91f8f514bf8bdab3857f0ca"} Nov 02 19:51:42.223571 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:42.223540 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:42.223571 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:42.223571 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:42.223770 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:42.223595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:42.223770 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.223691 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:42.223770 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.223701 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:42.223770 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.223741 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs podName:cd701aaf-13ea-4699-9c01-5167947a614a nodeName:}" failed. No retries permitted until 2026-11-02 19:51:58.223724568 +0000 UTC m=+34.312063468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs") pod "network-metrics-daemon-7nt8t" (UID: "cd701aaf-13ea-4699-9c01-5167947a614a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:42.223770 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.223768 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret podName:76aa7437-fa4a-42a6-90d5-9043d9b8a71d nodeName:}" failed. No retries permitted until 2026-11-02 19:51:58.223749124 +0000 UTC m=+34.312088015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret") pod "global-pull-secret-syncer-wkg2t" (UID: "76aa7437-fa4a-42a6-90d5-9043d9b8a71d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:42.223770 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.223709 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:42.223977 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.223794 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:42.223977 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.223808 2581 projected.go:196] Error preparing data for projected volume kube-api-access-psrcq for pod openshift-network-diagnostics/network-check-target-zg4fd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:42.223977 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.223844 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq podName:040d6f5e-129c-4665-ad54-957415093fa3 nodeName:}" failed. No retries permitted until 2026-11-02 19:51:58.223834521 +0000 UTC m=+34.312173408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-psrcq" (UniqueName: "kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq") pod "network-check-target-zg4fd" (UID: "040d6f5e-129c-4665-ad54-957415093fa3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:42.455193 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:42.455173 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:42.455311 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:42.455192 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:42.455311 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:42.455256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:42.455404 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.455387 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:42.455488 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.455469 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:42.455574 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:42.455557 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:42.594699 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:42.594478 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" event={"ID":"f81e0175-b8d3-4348-bc49-d9e00531643f","Type":"ContainerStarted","Data":"cbb92df92d9fe002bf277242711f07ea8e281c922b35bfa72186ae8c06ecf60d"} Nov 02 19:51:42.595083 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:42.594765 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" event={"ID":"f81e0175-b8d3-4348-bc49-d9e00531643f","Type":"ContainerStarted","Data":"ed94f2218ef46325d261b775db6640bcfb4cea22220a9920201ee43b89bd1fd2"} Nov 02 19:51:42.595083 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:42.594821 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" event={"ID":"f81e0175-b8d3-4348-bc49-d9e00531643f","Type":"ContainerStarted","Data":"9ac8ef4a23d8546020b1a8987e2597a2a20f6c551ed9ef0eff49053f19958d69"} Nov 02 19:51:43.600997 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:43.600963 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" event={"ID":"f81e0175-b8d3-4348-bc49-d9e00531643f","Type":"ContainerStarted","Data":"cd335432fcd24113212c763183d36175a2d8226272f67fbf7d72afe385025ca5"} Nov 02 19:51:43.601452 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:43.601084 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" event={"ID":"f81e0175-b8d3-4348-bc49-d9e00531643f","Type":"ContainerStarted","Data":"f21ecafa4112bcf728d9b144a47265f6c4da1d5e61bb1ee61b794ce57f46b607"} Nov 02 19:51:43.601452 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:43.601134 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" event={"ID":"f81e0175-b8d3-4348-bc49-d9e00531643f","Type":"ContainerStarted","Data":"b76d18e6239590ef86b5d7039b60157adea43961fc9212c549124e5c20f75c5d"} Nov 02 19:51:44.456070 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:44.456033 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:44.456249 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:44.456116 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:44.456249 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:44.456209 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:44.456350 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:44.456311 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:44.456395 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:44.456356 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:44.456427 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:44.456412 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:44.606145 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:44.606108 2581 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="2b76b5a6fd18e447f80b09bde6b7f8df7623948983d8f11f002d980b2f314255" exitCode=0 Nov 02 19:51:44.606633 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:44.606156 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" event={"ID":"6048b8aa-9e93-4214-8b5a-ed6663b632b5","Type":"ContainerDied","Data":"2b76b5a6fd18e447f80b09bde6b7f8df7623948983d8f11f002d980b2f314255"} Nov 02 19:51:45.614060 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:45.614028 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" event={"ID":"f81e0175-b8d3-4348-bc49-d9e00531643f","Type":"ContainerStarted","Data":"1348aad7b5a0dea3742a4b46dd68b046a1691d4b9b2c2c314f38861c6b9e7d07"} Nov 02 19:51:45.616810 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:45.616783 2581 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="0fe98755723a0c3efbb9115265c1c33e6b9434c73012b87deb48f1a956621b28" exitCode=0 Nov 02 19:51:45.616939 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:45.616827 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" event={"ID":"6048b8aa-9e93-4214-8b5a-ed6663b632b5","Type":"ContainerDied","Data":"0fe98755723a0c3efbb9115265c1c33e6b9434c73012b87deb48f1a956621b28"} Nov 02 19:51:46.455542 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:46.455506 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:46.455737 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:46.455506 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:46.455737 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:46.455606 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:46.455737 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:46.455712 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:46.455737 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:46.455521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:46.455954 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:46.455789 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:46.621587 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:46.621554 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" event={"ID":"6048b8aa-9e93-4214-8b5a-ed6663b632b5","Type":"ContainerStarted","Data":"584282e886282175e3c3c1ad8306a69206b9298a8738bf16afa90a5128dd3452"} Nov 02 19:51:46.738626 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:46.738517 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bn5zl" podStartSLOduration=8.246594749 podStartE2EDuration="21.738502955s" totalImagesPullingTime="13.491908206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:46.737988302 +0000 UTC m=+22.826327210" watchObservedRunningTime="2026-11-02 19:51:46.738502955 +0000 UTC m=+22.826841862" Nov 02 19:51:47.627750 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:47.627566 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" event={"ID":"f81e0175-b8d3-4348-bc49-d9e00531643f","Type":"ContainerStarted","Data":"c30d65cc9cacb12491b63bf328cbf2d542f78bac07b060e17a90a8657128070e"} Nov 02 19:51:47.898674 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:47.898557 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" podStartSLOduration=7.230073911 podStartE2EDuration="22.898544737s" totalImagesPullingTime="15.668470826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:51:47.847873005 +0000 UTC m=+23.936211913" watchObservedRunningTime="2026-11-02 19:51:47.898544737 +0000 UTC m=+23.986883644" Nov 02 19:51:48.455450 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:48.455404 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:48.455747 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:48.455422 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:48.455747 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:48.455524 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:48.455747 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:48.455533 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:48.455747 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:48.455598 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:48.455747 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:48.455669 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:48.630297 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:48.630253 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:51:48.630705 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:48.630686 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:48.630819 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:48.630776 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:48.646711 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:48.646683 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:48.646853 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:48.646821 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:49.540685 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:49.540606 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wkg2t"] Nov 02 19:51:49.540832 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:49.540722 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:49.540873 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:49.540821 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:49.543035 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:49.543007 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7nt8t"] Nov 02 19:51:49.543154 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:49.543114 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:49.543237 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:49.543218 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:49.543609 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:49.543594 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zg4fd"] Nov 02 19:51:49.543663 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:49.543654 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:49.543727 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:49.543714 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:49.632862 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:49.632833 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:51:50.635182 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:50.635153 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:51:51.166694 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:51.166593 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:51:51.177354 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:51.177315 2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" podUID="f81e0175-b8d3-4348-bc49-d9e00531643f" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 19:51:51.185817 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:51.185778 2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" podUID="f81e0175-b8d3-4348-bc49-d9e00531643f" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 19:51:51.454968 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:51.454878 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:51.454968 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:51.454937 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:51.454968 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:51.454934 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:51.455217 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:51.455034 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:51.455217 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:51.455153 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:51.455340 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:51.455222 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:51.646974 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:51.646942 2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" podUID="f81e0175-b8d3-4348-bc49-d9e00531643f" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 19:51:51.859652 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:51.859624 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nc9rw_1e35ec0a-ad15-48bc-ac59-e08c8dec75a0/dns-node-resolver/0.log" Nov 02 19:51:52.504783 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:52.504753 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rmpzb_b39d4538-dc20-4f6e-8170-4f5c0327cf69/node-ca/0.log" Nov 02 19:51:53.455232 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:53.455199 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:53.455723 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:53.455320 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:53.455723 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:53.455324 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:53.455723 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:53.455327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:53.455723 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:53.455430 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:53.455723 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:53.455515 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:55.455410 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:55.455187 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:55.455777 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:55.455217 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:55.455777 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:55.455495 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:55.455777 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:55.455187 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:55.455777 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:55.455556 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:55.455777 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:55.455640 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:57.455176 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:57.455137 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:57.455176 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:57.455174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:57.455720 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:57.455138 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:57.455720 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:57.455292 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:57.455720 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:57.455334 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:57.455720 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:57.455395 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:51:58.237778 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:58.237744 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:58.237962 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:58.237786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:58.237962 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:58.237846 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:58.237962 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:58.237915 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:51:58.237962 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:58.237940 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:51:58.237962 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:58.237950 2581 projected.go:196] Error preparing data for projected volume kube-api-access-psrcq for pod openshift-network-diagnostics/network-check-target-zg4fd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:58.238113 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:58.237960 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:58.238113 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:58.237994 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq podName:040d6f5e-129c-4665-ad54-957415093fa3 nodeName:}" failed. No retries permitted until 2026-11-02 19:52:30.237978393 +0000 UTC m=+66.326317293 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-psrcq" (UniqueName: "kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq") pod "network-check-target-zg4fd" (UID: "040d6f5e-129c-4665-ad54-957415093fa3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:51:58.238113 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:58.237919 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:58.238113 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:58.238012 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs podName:cd701aaf-13ea-4699-9c01-5167947a614a nodeName:}" failed. No retries permitted until 2026-11-02 19:52:30.238003514 +0000 UTC m=+66.326342408 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs") pod "network-metrics-daemon-7nt8t" (UID: "cd701aaf-13ea-4699-9c01-5167947a614a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:51:58.238113 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:58.238040 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret podName:76aa7437-fa4a-42a6-90d5-9043d9b8a71d nodeName:}" failed. No retries permitted until 2026-11-02 19:52:30.238030815 +0000 UTC m=+66.326369705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret") pod "global-pull-secret-syncer-wkg2t" (UID: "76aa7437-fa4a-42a6-90d5-9043d9b8a71d") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:51:59.454977 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:59.454942 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:51:59.454977 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:59.454977 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:51:59.455611 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:59.455052 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:51:59.455611 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:51:59.455106 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:51:59.455611 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:59.455194 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:51:59.455611 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:51:59.455271 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:52:01.455401 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:01.455363 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:01.455802 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:01.455419 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:01.455802 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:01.455382 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:01.455802 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:01.455473 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:52:01.455802 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:01.455556 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:52:01.455802 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:01.455622 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:52:03.455296 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:03.455247 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:03.455704 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:03.455247 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:03.455704 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:03.455387 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:52:03.455704 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:03.455247 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:03.455704 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:03.455508 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:52:03.455704 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:03.455427 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:52:05.454884 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:05.454846 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:05.454884 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:05.454880 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:05.455429 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:05.454988 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:52:05.455429 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:05.455031 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:05.455429 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:05.455013 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:52:05.455429 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:05.455096 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:52:07.455195 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:07.455166 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:07.455579 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:07.455166 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:07.455579 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:07.455293 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:52:07.455579 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:07.455166 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:07.455579 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:07.455368 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:52:07.455871 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:07.455841 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:52:09.455356 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:09.455316 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:09.455749 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:09.455371 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:09.455749 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:09.455484 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7nt8t" podUID="cd701aaf-13ea-4699-9c01-5167947a614a" Nov 02 19:52:09.455749 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:09.455537 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:09.455749 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:09.455607 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wkg2t" podUID="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" Nov 02 19:52:09.455749 ip-10-0-130-111 kubenswrapper[2581]: E1102 19:52:09.455669 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zg4fd" podUID="040d6f5e-129c-4665-ad54-957415093fa3" Nov 02 19:52:11.226530 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.226495 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeReady" Nov 02 19:52:11.226999 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.226655 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 19:52:11.379781 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.379705 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-stttr"] Nov 02 19:52:11.381591 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.381566 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:11.389814 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.389797 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pst9l"] Nov 02 19:52:11.391424 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.391410 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-stttr"] Nov 02 19:52:11.391479 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.391436 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pst9l"] Nov 02 19:52:11.391529 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.391517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.422714 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.422687 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-87h2b"] Nov 02 19:52:11.424484 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.424470 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.433898 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.433867 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:52:11.433898 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.433867 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:52:11.434063 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.433912 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:52:11.434063 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.433932 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d5pq8\"" Nov 02 19:52:11.443707 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.443692 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:52:11.443862 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.443846 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:52:11.443933 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.443872 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7qs77\"" Nov 02 19:52:11.443933 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.443922 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:52:11.444012 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.443952 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:52:11.455377 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.455359 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:11.455430 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.455419 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:11.455464 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.455442 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:11.478232 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.478211 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-87h2b"] Nov 02 19:52:11.522064 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522026 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36ec01c3-dcbe-4caf-baff-dcade7691e2e-crio-socket\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.522235 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522069 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-config-volume\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.522235 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522093 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-tmp-dir\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.522235 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522163 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wfgp\" (UniqueName: \"kubernetes.io/projected/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-kube-api-access-6wfgp\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.522424 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522269 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.522424 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522346 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36ec01c3-dcbe-4caf-baff-dcade7691e2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.522424 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71685c84-43a3-4272-8d8b-d6c623f8f592-cert\") pod \"ingress-canary-stttr\" (UID: \"71685c84-43a3-4272-8d8b-d6c623f8f592\") " pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:11.522424 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-metrics-tls\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.522554 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36ec01c3-dcbe-4caf-baff-dcade7691e2e-data-volume\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.522554 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhzfq\" (UniqueName: \"kubernetes.io/projected/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-api-access-nhzfq\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.522554 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.522525 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2f7\" (UniqueName: \"kubernetes.io/projected/71685c84-43a3-4272-8d8b-d6c623f8f592-kube-api-access-9s2f7\") pod \"ingress-canary-stttr\" (UID: \"71685c84-43a3-4272-8d8b-d6c623f8f592\") " pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:11.534397 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.534374 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvmjq\"" Nov 02 19:52:11.534520 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.534398 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:52:11.534582 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.534536 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:52:11.534699 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.534678 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gf4xf\"" Nov 02 19:52:11.534699 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.534703 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:52:11.534832 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.534717 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:52:11.534832 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.534731 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:52:11.534832 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.534751 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ltmn5\"" Nov 02 19:52:11.534832 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.534754 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sk47\"" Nov 02 19:52:11.534832 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.534775 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:52:11.623122 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.623122 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36ec01c3-dcbe-4caf-baff-dcade7691e2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.623367 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623142 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71685c84-43a3-4272-8d8b-d6c623f8f592-cert\") pod \"ingress-canary-stttr\" (UID: \"71685c84-43a3-4272-8d8b-d6c623f8f592\") " pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:11.623367 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-metrics-tls\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.623479 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623383 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36ec01c3-dcbe-4caf-baff-dcade7691e2e-data-volume\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.623479 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzfq\" (UniqueName: \"kubernetes.io/projected/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-api-access-nhzfq\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.623479 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623469 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2f7\" (UniqueName: \"kubernetes.io/projected/71685c84-43a3-4272-8d8b-d6c623f8f592-kube-api-access-9s2f7\") pod \"ingress-canary-stttr\" (UID: \"71685c84-43a3-4272-8d8b-d6c623f8f592\") " pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:11.623615 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623525 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36ec01c3-dcbe-4caf-baff-dcade7691e2e-crio-socket\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.623615 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-config-volume\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.623615 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-tmp-dir\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.623615 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623599 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wfgp\" (UniqueName: \"kubernetes.io/projected/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-kube-api-access-6wfgp\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.623787 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623715 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36ec01c3-dcbe-4caf-baff-dcade7691e2e-data-volume\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.624400 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623845 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.624400 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.623871 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36ec01c3-dcbe-4caf-baff-dcade7691e2e-crio-socket\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.624400 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.624162 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-tmp-dir\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.624400 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.624359 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-config-volume\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.626906 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.626884 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-metrics-tls\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.626988 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.626937 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36ec01c3-dcbe-4caf-baff-dcade7691e2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.627221 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.627204 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71685c84-43a3-4272-8d8b-d6c623f8f592-cert\") pod \"ingress-canary-stttr\" (UID: \"71685c84-43a3-4272-8d8b-d6c623f8f592\") " pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:11.680318 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.680240 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wfgp\" (UniqueName: \"kubernetes.io/projected/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-kube-api-access-6wfgp\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:11.680417 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.680335 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzfq\" (UniqueName: \"kubernetes.io/projected/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-api-access-nhzfq\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.680464 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.680449 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2f7\" (UniqueName: \"kubernetes.io/projected/71685c84-43a3-4272-8d8b-d6c623f8f592-kube-api-access-9s2f7\") pod \"ingress-canary-stttr\" (UID: \"71685c84-43a3-4272-8d8b-d6c623f8f592\") " pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:11.693126 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.693105 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:11.704918 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.704891 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:11.734749 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:11.734717 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:12.075457 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:12.075430 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-stttr"] Nov 02 19:52:12.075686 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:52:12.075663 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71685c84_43a3_4272_8d8b_d6c623f8f592.slice/crio-4fd3a04e4b6785a304f94c2ec8aab79af236b59e07d73f10b441dff3fdc0c28d WatchSource:0}: Error finding container 4fd3a04e4b6785a304f94c2ec8aab79af236b59e07d73f10b441dff3fdc0c28d: Status 404 returned error can't find the container with id 4fd3a04e4b6785a304f94c2ec8aab79af236b59e07d73f10b441dff3fdc0c28d Nov 02 19:52:12.077200 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:52:12.077179 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ec01c3_dcbe_4caf_baff_dcade7691e2e.slice/crio-cf9c00cc60cedf8e413bbcf47d6a2dcd19eafc60b48025d55bdd5a4f8cbc6ca3 WatchSource:0}: Error finding container cf9c00cc60cedf8e413bbcf47d6a2dcd19eafc60b48025d55bdd5a4f8cbc6ca3: Status 404 returned error can't find the container with id cf9c00cc60cedf8e413bbcf47d6a2dcd19eafc60b48025d55bdd5a4f8cbc6ca3 Nov 02 19:52:12.078966 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:12.078946 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pst9l"] Nov 02 19:52:12.119580 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:52:12.119554 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfeeb7e_97f0_4d57_b450_bd17b4dfc8b1.slice/crio-4c390dfedefcfa015389b92b41df59eadbb0341df41d633456a2746c4b7c6c90 WatchSource:0}: Error finding container 4c390dfedefcfa015389b92b41df59eadbb0341df41d633456a2746c4b7c6c90: Status 404 returned error can't find the container with id 4c390dfedefcfa015389b92b41df59eadbb0341df41d633456a2746c4b7c6c90 Nov 02 19:52:12.121327 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:12.121302 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-87h2b"] Nov 02 19:52:12.681651 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:12.681617 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-87h2b" event={"ID":"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1","Type":"ContainerStarted","Data":"4c390dfedefcfa015389b92b41df59eadbb0341df41d633456a2746c4b7c6c90"} Nov 02 19:52:12.683127 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:12.683105 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pst9l" event={"ID":"36ec01c3-dcbe-4caf-baff-dcade7691e2e","Type":"ContainerStarted","Data":"15cea2427704adee162cc30377edce194904194b3ddea8024f211326109b78ba"} Nov 02 19:52:12.683232 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:12.683149 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pst9l" event={"ID":"36ec01c3-dcbe-4caf-baff-dcade7691e2e","Type":"ContainerStarted","Data":"cf9c00cc60cedf8e413bbcf47d6a2dcd19eafc60b48025d55bdd5a4f8cbc6ca3"} Nov 02 19:52:12.684170 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:12.684149 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-stttr" event={"ID":"71685c84-43a3-4272-8d8b-d6c623f8f592","Type":"ContainerStarted","Data":"4fd3a04e4b6785a304f94c2ec8aab79af236b59e07d73f10b441dff3fdc0c28d"} Nov 02 19:52:13.690849 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:13.690805 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pst9l" event={"ID":"36ec01c3-dcbe-4caf-baff-dcade7691e2e","Type":"ContainerStarted","Data":"226ae0931985a6c82fe38a4dd9d0a3869828762b5fe9aa06a149c526450498f9"} Nov 02 19:52:14.695515 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:14.695475 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-87h2b" event={"ID":"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1","Type":"ContainerStarted","Data":"210ec3bebcdd3de5987ceb4c3da731ac58874e60fedb82f6354e3dcd460e9ccb"} Nov 02 19:52:14.695973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:14.695532 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-87h2b" event={"ID":"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1","Type":"ContainerStarted","Data":"400ce3fa3d4b13cb6cc46d019bb58cc4a70b722bc98b434648e8c95f0a4c180e"} Nov 02 19:52:14.695973 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:14.695627 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:14.813251 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:14.813195 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-87h2b" podStartSLOduration=1.6725105980000001 podStartE2EDuration="3.813181033s" totalImagesPullingTime="2.140670435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:14.812554913 +0000 UTC m=+50.900893822" watchObservedRunningTime="2026-11-02 19:52:14.813181033 +0000 UTC m=+50.901519940" Nov 02 19:52:15.077000 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.076962 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g"] Nov 02 19:52:15.081393 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.081364 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.133830 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.133748 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g"] Nov 02 19:52:15.183035 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.182989 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:52:15.183204 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.183051 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 19:52:15.183204 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.183058 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qc22z\"" Nov 02 19:52:15.184975 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.184948 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 19:52:15.184975 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.184968 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:52:15.185135 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.184989 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 19:52:15.253902 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.253873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.254007 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.253935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.254073 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.254010 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.254129 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.254088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqvgg\" (UniqueName: \"kubernetes.io/projected/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-kube-api-access-pqvgg\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.354785 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.354708 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.354785 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.354766 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.354992 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.354802 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvgg\" (UniqueName: \"kubernetes.io/projected/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-kube-api-access-pqvgg\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.354992 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.354881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.355667 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.355643 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.357354 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.357328 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.357524 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.357504 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.412227 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.412159 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvgg\" (UniqueName: \"kubernetes.io/projected/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-kube-api-access-pqvgg\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.697521 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.697460 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:15.699961 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.699879 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pst9l" event={"ID":"36ec01c3-dcbe-4caf-baff-dcade7691e2e","Type":"ContainerStarted","Data":"9e75fec695dd129c5e5e834d9df47183840d2555c5f4bad3ef5c9f9695dbe17b"} Nov 02 19:52:15.815201 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:15.815061 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pst9l" podStartSLOduration=1.79028608 podStartE2EDuration="4.81504375s" totalImagesPullingTime="3.02475767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:52:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:15.81455172 +0000 UTC m=+51.902890627" watchObservedRunningTime="2026-11-02 19:52:15.81504375 +0000 UTC m=+51.903382659" Nov 02 19:52:16.074063 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:52:16.074035 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6c8be3_97cb_4ac9_be0e_b6bf3fff4a98.slice/crio-1ff7077a7fe69feb99960b9c6b29819245a604dfe3769ad309b348b089f86664 WatchSource:0}: Error finding container 1ff7077a7fe69feb99960b9c6b29819245a604dfe3769ad309b348b089f86664: Status 404 returned error can't find the container with id 1ff7077a7fe69feb99960b9c6b29819245a604dfe3769ad309b348b089f86664 Nov 02 19:52:16.074679 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:16.074631 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g"] Nov 02 19:52:16.703249 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:16.703215 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-stttr" event={"ID":"71685c84-43a3-4272-8d8b-d6c623f8f592","Type":"ContainerStarted","Data":"3e1f7d857bedf44fc30eaad4944422e9cd087a551b486179820bc6e8f87d4988"} Nov 02 19:52:16.704397 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:16.704373 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" event={"ID":"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98","Type":"ContainerStarted","Data":"1ff7077a7fe69feb99960b9c6b29819245a604dfe3769ad309b348b089f86664"} Nov 02 19:52:16.817192 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:16.817137 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-stttr" podStartSLOduration=2.1688809510000002 podStartE2EDuration="5.817121493s" totalImagesPullingTime="3.648240542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:16.816360455 +0000 UTC m=+52.904699362" watchObservedRunningTime="2026-11-02 19:52:16.817121493 +0000 UTC m=+52.905460400" Nov 02 19:52:18.713675 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:18.713641 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" event={"ID":"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98","Type":"ContainerStarted","Data":"138328f10670dc47a57d4b1fa2eaa7a95380864554eb2ffd7a3118add6d97f6b"} Nov 02 19:52:18.714189 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:18.713694 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" event={"ID":"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98","Type":"ContainerStarted","Data":"688d4abb50784dacbe092a42d310e2f3d952aa2d9816ae60df47a0a9d7326a6f"} Nov 02 19:52:18.829033 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:18.828977 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" podStartSLOduration=2.107495872 podStartE2EDuration="3.828963361s" totalImagesPullingTime="1.721467489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:18.828599422 +0000 UTC m=+54.916938330" watchObservedRunningTime="2026-11-02 19:52:18.828963361 +0000 UTC m=+54.917302306" Nov 02 19:52:21.647650 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.647622 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:21.860182 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.860148 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x4ttn"] Nov 02 19:52:21.863386 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.863366 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:21.909012 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.908939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-sys\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:21.909012 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.908975 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gbmv\" (UniqueName: \"kubernetes.io/projected/002bc527-a392-444a-85a5-2dbe005559f4-kube-api-access-6gbmv\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:21.909012 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.908998 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-tls\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:21.909250 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.909018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-metrics-client-ca\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:21.909250 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.909072 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:21.909250 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.909213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-root\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:21.909250 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.909239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-textfile\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:21.909462 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.909323 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:21.909462 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.909346 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-wtmp\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:21.922473 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.922444 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 19:52:21.922620 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.922477 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 19:52:21.922620 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.922446 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 19:52:21.922620 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:21.922523 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xjzch\"" Nov 02 19:52:22.010054 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-root\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010054 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010048 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-textfile\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010330 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010135 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-root\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010330 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010175 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010330 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010207 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-wtmp\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010330 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-sys\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010529 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010337 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-wtmp\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010529 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbmv\" (UniqueName: \"kubernetes.io/projected/002bc527-a392-444a-85a5-2dbe005559f4-kube-api-access-6gbmv\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010529 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010376 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-sys\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010529 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010383 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-tls\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010529 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-metrics-client-ca\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010529 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010434 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010723 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010532 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-textfile\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010855 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010823 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.010978 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.010927 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-metrics-client-ca\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.012590 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.012564 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-tls\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.012683 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.012652 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.069240 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.069208 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbmv\" (UniqueName: \"kubernetes.io/projected/002bc527-a392-444a-85a5-2dbe005559f4-kube-api-access-6gbmv\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.178169 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.178085 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:22.187749 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:52:22.187712 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002bc527_a392_444a_85a5_2dbe005559f4.slice/crio-8b4e7d5968a48e4328cc58f12cbcf6796fd18e4cbfd314d6894474b1dca4a64f WatchSource:0}: Error finding container 8b4e7d5968a48e4328cc58f12cbcf6796fd18e4cbfd314d6894474b1dca4a64f: Status 404 returned error can't find the container with id 8b4e7d5968a48e4328cc58f12cbcf6796fd18e4cbfd314d6894474b1dca4a64f Nov 02 19:52:22.725151 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:22.725115 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4ttn" event={"ID":"002bc527-a392-444a-85a5-2dbe005559f4","Type":"ContainerStarted","Data":"8b4e7d5968a48e4328cc58f12cbcf6796fd18e4cbfd314d6894474b1dca4a64f"} Nov 02 19:52:23.979404 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:23.979319 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:52:23.982571 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:23.982556 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.027745 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.027714 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.027880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.027756 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-out\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.027880 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.027850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.028007 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.027912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbxnx\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-kube-api-access-vbxnx\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.028007 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.027946 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.028105 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.028024 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.028105 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.028052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.028105 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.028075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.028200 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.028149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-web-config\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.028200 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.028175 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.028200 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.028190 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.028333 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.028213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-volume\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.028333 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.028234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.030885 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.030859 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:52:24.080298 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.080248 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 19:52:24.080298 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.080258 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 19:52:24.080512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.080306 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 19:52:24.080512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.080322 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 19:52:24.080512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.080376 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 19:52:24.080512 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.080440 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ks6xc\"" Nov 02 19:52:24.080779 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.080750 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 19:52:24.129288 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129453 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-out\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129453 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129344 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129453 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129364 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxnx\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-kube-api-access-vbxnx\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129453 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129600 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129498 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129600 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129524 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129600 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129600 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129583 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-web-config\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129779 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129613 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129779 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129636 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129779 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129672 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-volume\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129779 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.129979 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.129933 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.130671 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.130646 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.132297 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.132249 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.132526 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.132480 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.132526 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.132508 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.132638 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.132550 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-out\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.132638 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.132627 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.132796 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.132776 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.132951 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.132932 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-volume\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.153705 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.153680 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 19:52:24.153705 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.153680 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 19:52:24.153855 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.153758 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 19:52:24.160912 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.160886 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.162077 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.162054 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-web-config\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.162197 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.162176 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.186078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.186056 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxnx\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-kube-api-access-vbxnx\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.360150 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.360121 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ks6xc\"" Nov 02 19:52:24.369211 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.369190 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:24.702001 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.701925 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:24.732450 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.732415 2581 generic.go:356] "Generic (PLEG): container finished" podID="002bc527-a392-444a-85a5-2dbe005559f4" containerID="0e195e598b6a0f38943e8df31d0589058f4e23dcafb44582fb4ba6b4e4297967" exitCode=0 Nov 02 19:52:24.732622 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.732458 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4ttn" event={"ID":"002bc527-a392-444a-85a5-2dbe005559f4","Type":"ContainerDied","Data":"0e195e598b6a0f38943e8df31d0589058f4e23dcafb44582fb4ba6b4e4297967"} Nov 02 19:52:24.836193 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:24.836167 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:52:24.881005 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:52:24.880977 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d53725_3f49_4c81_b68a_1a60a69a2fbc.slice/crio-26e67e5f5480c8e90868081c6f4fc876dd75f50e375542abc6d78ced2ffa2828 WatchSource:0}: Error finding container 26e67e5f5480c8e90868081c6f4fc876dd75f50e375542abc6d78ced2ffa2828: Status 404 returned error can't find the container with id 26e67e5f5480c8e90868081c6f4fc876dd75f50e375542abc6d78ced2ffa2828 Nov 02 19:52:25.737597 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:25.737559 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4ttn" event={"ID":"002bc527-a392-444a-85a5-2dbe005559f4","Type":"ContainerStarted","Data":"6374dad75339a5b8b9edd4293801f88c4ca8f501a6a6e23b633bad095708c71d"} Nov 02 19:52:25.738059 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:25.737618 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4ttn" event={"ID":"002bc527-a392-444a-85a5-2dbe005559f4","Type":"ContainerStarted","Data":"26eb19bfe06edb79e736b229522dbd260c8545cf28ddbea41e5f49048853cf56"} Nov 02 19:52:25.738812 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:25.738788 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerStarted","Data":"26e67e5f5480c8e90868081c6f4fc876dd75f50e375542abc6d78ced2ffa2828"} Nov 02 19:52:25.856192 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:25.856137 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x4ttn" podStartSLOduration=3.25808788 podStartE2EDuration="4.856119411s" totalImagesPullingTime="1.53178888s" totalInitContainerRuntime="66.242651ms" isStatefulPod=true podCreationTimestamp="2026-11-02 19:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:25.853819692 +0000 UTC m=+61.942158600" watchObservedRunningTime="2026-11-02 19:52:25.856119411 +0000 UTC m=+61.944458345" Nov 02 19:52:26.743682 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:26.743593 2581 generic.go:356] "Generic (PLEG): container finished" podID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerID="3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c" exitCode=0 Nov 02 19:52:26.744130 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:26.743685 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerDied","Data":"3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c"} Nov 02 19:52:29.757754 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:29.757712 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerStarted","Data":"b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db"} Nov 02 19:52:29.758249 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:29.757794 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerStarted","Data":"24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227"} Nov 02 19:52:29.758249 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:29.757833 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerStarted","Data":"6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a"} Nov 02 19:52:29.758249 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:29.757872 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerStarted","Data":"fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0"} Nov 02 19:52:29.758249 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:29.757914 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerStarted","Data":"9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5"} Nov 02 19:52:30.279887 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.279849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:30.280166 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.279895 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:30.280166 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.279928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:30.382360 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.382333 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:52:30.382633 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.382525 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:52:30.382633 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.382559 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:52:30.393638 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.393615 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:30.397642 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.397617 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:30.442223 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.442194 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:52:30.454063 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.454027 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:30.718423 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.718393 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sk47\"" Nov 02 19:52:30.723385 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.723369 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gf4xf\"" Nov 02 19:52:30.727358 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.727341 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:30.727612 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.727600 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ltmn5\"" Nov 02 19:52:30.732201 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.732185 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:30.737162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.737132 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:30.788529 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.788506 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerStarted","Data":"507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e"} Nov 02 19:52:30.862240 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.862192 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 19:52:30.866903 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.866887 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.920301 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.920189 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r6f2h\"" Nov 02 19:52:30.920911 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.920866 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.8684446489999997 podStartE2EDuration="7.920851417s" totalImagesPullingTime="5.039454945s" totalInitContainerRuntime="12.951823ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:30.920483419 +0000 UTC m=+67.008822326" watchObservedRunningTime="2026-11-02 19:52:30.920851417 +0000 UTC m=+67.009190325" Nov 02 19:52:30.947236 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.947202 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 19:52:30.947236 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.947216 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 19:52:30.947236 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.947240 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 19:52:30.947509 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.947216 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 19:52:30.985568 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.985568 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985571 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.985787 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985590 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.985787 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985658 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.985887 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985785 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.985887 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985818 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.985887 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.985971 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985894 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.985971 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.985971 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985958 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.986081 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.985976 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.986081 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.986026 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.986138 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.986099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.986177 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.986137 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.986177 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.986170 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config-out\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.986250 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.986194 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.986250 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.986217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-web-config\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.986355 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.986257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.986398 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.986371 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:30.986438 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:30.986399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjk47\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-kube-api-access-zjk47\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021464 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021491 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021490 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021464 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021492 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021542 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021557 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021590 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021597 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021591 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 19:52:31.021634 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021632 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 19:52:31.022222 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.021671 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2uh2tiljffmk1\"" Nov 02 19:52:31.022731 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.022713 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 19:52:31.087522 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087487 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-web-config\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.087522 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.087754 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087543 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.087754 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087563 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjk47\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-kube-api-access-zjk47\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.087754 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.087754 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087619 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.087754 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.087754 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087713 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.087754 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087794 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087819 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087892 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087970 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.087996 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config-out\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088078 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.088032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088616 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.088467 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088616 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.088507 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.088724 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.088664 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.090709 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.088951 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.090709 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.089217 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.090885 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.089334 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.090885 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.090644 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.091234 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.091209 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.092162 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.092127 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.093142 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.093061 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.093142 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.093092 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.093142 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.093107 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-web-config\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.093371 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.093262 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.093932 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.093899 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config-out\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.094076 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.094052 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.094385 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.094362 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.094894 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.094869 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.095236 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.095214 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.095298 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.095268 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.107410 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.107388 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wkg2t"] Nov 02 19:52:31.107501 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:52:31.107483 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd701aaf_13ea_4699_9c01_5167947a614a.slice/crio-dada2124e1ccdfdbbede798a85d63a3031ded078504441eab2b3c8b52f94fbd0 WatchSource:0}: Error finding container dada2124e1ccdfdbbede798a85d63a3031ded078504441eab2b3c8b52f94fbd0: Status 404 returned error can't find the container with id dada2124e1ccdfdbbede798a85d63a3031ded078504441eab2b3c8b52f94fbd0 Nov 02 19:52:31.108980 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.108959 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7nt8t"] Nov 02 19:52:31.143389 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:52:31.143356 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040d6f5e_129c_4665_ad54_957415093fa3.slice/crio-e188059c884916b475a8da5131e210badc00b011f36195e3b4313c88ad7f9532 WatchSource:0}: Error finding container e188059c884916b475a8da5131e210badc00b011f36195e3b4313c88ad7f9532: Status 404 returned error can't find the container with id e188059c884916b475a8da5131e210badc00b011f36195e3b4313c88ad7f9532 Nov 02 19:52:31.144541 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.144521 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zg4fd"] Nov 02 19:52:31.147197 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.147177 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjk47\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-kube-api-access-zjk47\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.196360 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.196244 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:31.610626 ip-10-0-130-111 kubenswrapper[2581]: W1102 19:52:31.610595 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f8744c_1c79_4c6a_b2fb_c7aeac335710.slice/crio-b3e47f32cb33b982fac5a898072a4791f9a1b54c4b50207a5c7587844802885c WatchSource:0}: Error finding container b3e47f32cb33b982fac5a898072a4791f9a1b54c4b50207a5c7587844802885c: Status 404 returned error can't find the container with id b3e47f32cb33b982fac5a898072a4791f9a1b54c4b50207a5c7587844802885c Nov 02 19:52:31.612238 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.612218 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 19:52:31.794695 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.794660 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zg4fd" event={"ID":"040d6f5e-129c-4665-ad54-957415093fa3","Type":"ContainerStarted","Data":"e188059c884916b475a8da5131e210badc00b011f36195e3b4313c88ad7f9532"} Nov 02 19:52:31.796041 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.796009 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7nt8t" event={"ID":"cd701aaf-13ea-4699-9c01-5167947a614a","Type":"ContainerStarted","Data":"dada2124e1ccdfdbbede798a85d63a3031ded078504441eab2b3c8b52f94fbd0"} Nov 02 19:52:31.797570 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.797548 2581 generic.go:356] "Generic (PLEG): container finished" podID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerID="edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded" exitCode=0 Nov 02 19:52:31.797676 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.797621 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerDied","Data":"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded"} Nov 02 19:52:31.797720 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.797675 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerStarted","Data":"b3e47f32cb33b982fac5a898072a4791f9a1b54c4b50207a5c7587844802885c"} Nov 02 19:52:31.799001 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:31.798975 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wkg2t" event={"ID":"76aa7437-fa4a-42a6-90d5-9043d9b8a71d","Type":"ContainerStarted","Data":"a8ca966ae355cc3d44fc2d72a49ecb5a6e94a29133fc59fef18ffc8150c125c0"} Nov 02 19:52:33.811953 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:33.811914 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7nt8t" event={"ID":"cd701aaf-13ea-4699-9c01-5167947a614a","Type":"ContainerStarted","Data":"c9f457630eceb6be03cd11de8ff2ccd989f99e5dc8e15f952e218607f3506a8d"} Nov 02 19:52:33.812432 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:33.811978 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7nt8t" event={"ID":"cd701aaf-13ea-4699-9c01-5167947a614a","Type":"ContainerStarted","Data":"aaffaf15c23080935a50efa9342983baf32cd992325c14ac12d7c2a92647dbd4"} Nov 02 19:52:34.035017 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:34.034962 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7nt8t" podStartSLOduration=67.184920087 podStartE2EDuration="1m9.034947313s" totalImagesPullingTime="1.850027226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:34.033605586 +0000 UTC m=+70.121944498" watchObservedRunningTime="2026-11-02 19:52:34.034947313 +0000 UTC m=+70.123286221" Nov 02 19:52:35.823023 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:35.822988 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zg4fd" event={"ID":"040d6f5e-129c-4665-ad54-957415093fa3","Type":"ContainerStarted","Data":"f5062e4f00e5b58541382e335fa6e60d1d73186f63196b2112a69e6d673a6566"} Nov 02 19:52:35.823452 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:35.823096 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:35.935489 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:35.935385 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zg4fd" podStartSLOduration=66.438491075 podStartE2EDuration="1m10.935370416s" totalImagesPullingTime="4.496879341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:35.935325548 +0000 UTC m=+72.023664457" watchObservedRunningTime="2026-11-02 19:52:35.935370416 +0000 UTC m=+72.023709323" Nov 02 19:52:36.829500 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:36.829461 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerStarted","Data":"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879"} Nov 02 19:52:36.829912 ip-10-0-130-111 kubenswrapper[2581]: I1102 19:52:36.829542 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerStarted","Data":"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080"} Nov 02 19:52:37.311758 ip-10-0-130-111 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 19:52:37.336701 ip-10-0-130-111 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 19:52:37.336903 ip-10-0-130-111 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 19:52:37.336958 ip-10-0-130-111 systemd[1]: kubelet.service: Consumed 3.832s CPU time. Nov 02 19:52:37.347809 ip-10-0-130-111 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:52:37.417661 ip-10-0-130-111 kubenswrapper[5399]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:52:37.417661 ip-10-0-130-111 kubenswrapper[5399]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:52:37.417661 ip-10-0-130-111 kubenswrapper[5399]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:52:37.417661 ip-10-0-130-111 kubenswrapper[5399]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:52:37.419681 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419602 5399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:37.419681 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419681 5399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:37.419681 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419685 5399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419688 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419691 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419694 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419697 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419699 5399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419703 5399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419705 5399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419710 5399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419713 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419715 5399 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419718 5399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419721 5399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419724 5399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419726 5399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419729 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419732 5399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419734 5399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419737 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419740 5399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:37.419780 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419742 5399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419745 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419748 5399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419750 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419753 5399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419756 5399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419758 5399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419761 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419763 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419766 5399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419769 5399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419773 5399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419778 5399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419781 5399 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419785 5399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419788 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419791 5399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419794 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419797 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:37.420269 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419800 5399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419802 5399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419805 5399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419808 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419810 5399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419813 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419815 5399 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419818 5399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419820 5399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419823 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419826 5399 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419829 5399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419832 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419834 5399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419837 5399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419840 5399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419842 5399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419845 5399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419848 5399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:37.420808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419850 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419853 5399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419855 5399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419858 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419861 5399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419863 5399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419866 5399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419870 5399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419875 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419879 5399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419883 5399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419887 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419891 5399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419895 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419899 5399 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419903 5399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419907 5399 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419911 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419915 5399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419921 5399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:37.421258 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419925 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419929 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419932 5399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419935 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419937 5399 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419940 5399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419943 5399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419946 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419949 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419952 5399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419954 5399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419957 5399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419960 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419963 5399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419965 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419968 5399 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419971 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419973 5399 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419977 5399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:37.421796 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419980 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419982 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419985 5399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419989 5399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419991 5399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419994 5399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.419997 5399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420828 5399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420836 5399 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420839 5399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420842 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420845 5399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420848 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420851 5399 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420853 5399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420856 5399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420859 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420862 5399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420864 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420867 5399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:37.422264 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420869 5399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420872 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420875 5399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420877 5399 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420881 5399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420883 5399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420886 5399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420888 5399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420891 5399 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420894 5399 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420899 5399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420904 5399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420907 5399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420911 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420913 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420916 5399 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420919 5399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420921 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420924 5399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420933 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:37.422774 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420935 5399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420938 5399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420941 5399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420944 5399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420946 5399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420949 5399 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420952 5399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420954 5399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420957 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420959 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420962 5399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420964 5399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420967 5399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420970 5399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420973 5399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420976 5399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420978 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420982 5399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420984 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:37.423266 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420987 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420990 5399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420993 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420996 5399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.420998 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421001 5399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421003 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421006 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421008 5399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421011 5399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421013 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421016 5399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421018 5399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421027 5399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421030 5399 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421033 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421035 5399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421038 5399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421041 5399 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421044 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:37.423734 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421046 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421049 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421051 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421054 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421056 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421059 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421062 5399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421065 5399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421068 5399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421070 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421073 5399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421076 5399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421078 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421081 5399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421083 5399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421086 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421089 5399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421092 5399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421095 5399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:37.424204 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421098 5399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421101 5399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421104 5399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421106 5399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421109 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421111 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421114 5399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421122 5399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421125 5399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421127 5399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421130 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421133 5399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421135 5399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421138 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421140 5399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421662 5399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421670 5399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421674 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421677 5399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421680 5399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:37.424683 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421683 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421686 5399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421688 5399 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421691 5399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421694 5399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421696 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421699 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421701 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421704 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421707 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421709 5399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421712 5399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421714 5399 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421717 5399 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421719 5399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421722 5399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421724 5399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421727 5399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421729 5399 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421732 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:37.425184 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421735 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421738 5399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421741 5399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421743 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421746 5399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421748 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421752 5399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421754 5399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421757 5399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421760 5399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421763 5399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421765 5399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421768 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421771 5399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421773 5399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421776 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421778 5399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421781 5399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421783 5399 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421786 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:37.425663 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421788 5399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421791 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421794 5399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421796 5399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421799 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421802 5399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421805 5399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421807 5399 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421810 5399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421813 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421816 5399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421819 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421822 5399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421824 5399 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421827 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421830 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421832 5399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421835 5399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421837 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421841 5399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:37.426179 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421843 5399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421846 5399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421848 5399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421851 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421853 5399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421856 5399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421858 5399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421862 5399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421866 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421869 5399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421873 5399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421876 5399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421879 5399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421882 5399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421885 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421889 5399 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421892 5399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421894 5399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421897 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:37.426698 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421899 5399 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421902 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421905 5399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421907 5399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421910 5399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421912 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421918 5399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421921 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421923 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421925 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421928 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421931 5399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421933 5399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421936 5399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421938 5399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421941 5399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421943 5399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421946 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421950 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:37.427236 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421952 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421955 5399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.421957 5399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422052 5399 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422066 5399 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422072 5399 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422077 5399 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422083 5399 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422087 5399 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422091 5399 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422095 5399 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422098 5399 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422101 5399 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422105 5399 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422108 5399 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422111 5399 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422114 5399 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422117 5399 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422120 5399 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422123 5399 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422127 5399 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422131 5399 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422134 5399 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422137 5399 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:52:37.428086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422140 5399 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422144 5399 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422148 5399 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422151 5399 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422154 5399 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422158 5399 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422163 5399 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422167 5399 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422170 5399 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422172 5399 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422176 5399 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422180 5399 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422183 5399 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422186 5399 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422189 5399 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422193 5399 flags.go:64] FLAG: --enable-server="true" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422196 5399 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422201 5399 flags.go:64] FLAG: --event-burst="100" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422204 5399 flags.go:64] FLAG: --event-qps="50" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422207 5399 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422210 5399 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422213 5399 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422217 5399 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422220 5399 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422223 5399 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:52:37.428924 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422226 5399 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422229 5399 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422232 5399 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422236 5399 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422239 5399 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422242 5399 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422245 5399 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422248 5399 flags.go:64] FLAG: --feature-gates="" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422252 5399 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422255 5399 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422258 5399 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422261 5399 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422265 5399 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422289 5399 flags.go:64] FLAG: --help="false" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422294 5399 flags.go:64] FLAG: --hostname-override="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422298 5399 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422302 5399 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422305 5399 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422308 5399 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422312 5399 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422315 5399 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422318 5399 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422322 5399 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:52:37.429553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422325 5399 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422328 5399 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422331 5399 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422334 5399 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422337 5399 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422340 5399 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422344 5399 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422346 5399 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422349 5399 flags.go:64] FLAG: --lock-file="" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422352 5399 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422355 5399 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422358 5399 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422369 5399 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422372 5399 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422376 5399 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422379 5399 flags.go:64] FLAG: --logging-format="text" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422382 5399 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422385 5399 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422388 5399 flags.go:64] FLAG: --manifest-url="" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422391 5399 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422395 5399 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422398 5399 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422403 5399 flags.go:64] FLAG: --max-pods="110" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422407 5399 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422410 5399 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:52:37.430128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422413 5399 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422416 5399 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422419 5399 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422422 5399 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422425 5399 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422433 5399 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422437 5399 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422440 5399 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422444 5399 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422447 5399 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422450 5399 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422453 5399 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422456 5399 flags.go:64] FLAG: --port="10250" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422459 5399 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422462 5399 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c8c2ef10f7847467" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422465 5399 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422468 5399 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422471 5399 flags.go:64] FLAG: --register-node="true" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422474 5399 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422478 5399 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422481 5399 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422484 5399 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422487 5399 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422492 5399 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422495 5399 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422498 5399 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:52:37.430827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422501 5399 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422504 5399 flags.go:64] FLAG: --runonce="false" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422506 5399 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422511 5399 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422514 5399 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422517 5399 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422520 5399 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422524 5399 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422527 5399 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422530 5399 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422533 5399 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422536 5399 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422540 5399 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422544 5399 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422547 5399 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422550 5399 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422553 5399 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422558 5399 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422561 5399 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422564 5399 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422568 5399 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422571 5399 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422574 5399 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422577 5399 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422580 5399 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:52:37.431478 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422583 5399 flags.go:64] FLAG: --v="2" Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422587 5399 flags.go:64] FLAG: --version="false" Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422591 5399 flags.go:64] FLAG: --vmodule="" Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422595 5399 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.422599 5399 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422731 5399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422734 5399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422737 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422740 5399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422743 5399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422747 5399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422750 5399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422752 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422755 5399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422758 5399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422761 5399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422764 5399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422767 5399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422770 5399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422774 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:37.432145 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422777 5399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422780 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422782 5399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422785 5399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422788 5399 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422790 5399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422793 5399 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422796 5399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422798 5399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422801 5399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422803 5399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422806 5399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422809 5399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422812 5399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422814 5399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422817 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422819 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422822 5399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422825 5399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422828 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:37.432804 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422830 5399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422833 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422836 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422839 5399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422841 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422844 5399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422846 5399 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422849 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422851 5399 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422855 5399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422860 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422866 5399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422869 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422871 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422874 5399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422877 5399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422879 5399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422882 5399 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422884 5399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422887 5399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:37.433475 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422889 5399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422892 5399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422894 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422897 5399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422899 5399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422902 5399 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422905 5399 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422907 5399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422910 5399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422912 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422915 5399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422918 5399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422921 5399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422923 5399 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422926 5399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422928 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422931 5399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422934 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422937 5399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422940 5399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:37.434186 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422944 5399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422947 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422950 5399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422953 5399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422956 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422959 5399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422962 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422965 5399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422967 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422970 5399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422973 5399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422975 5399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422978 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422981 5399 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422983 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422986 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422988 5399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422991 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422994 5399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:37.434687 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422996 5399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.422999 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423001 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423004 5399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423007 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423009 5399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423012 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423015 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423018 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423020 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423023 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423026 5399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.423051 5399 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423086 5399 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423090 5399 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423093 5399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:37.435182 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423096 5399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423100 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423103 5399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423106 5399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423108 5399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423111 5399 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423115 5399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423117 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423120 5399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423122 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423125 5399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423129 5399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423133 5399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423136 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423139 5399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423142 5399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423144 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423147 5399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423150 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:37.435675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423153 5399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423155 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423158 5399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423161 5399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423164 5399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423167 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423169 5399 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423172 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423175 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423178 5399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423180 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423183 5399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423185 5399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423188 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423192 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423194 5399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423197 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423200 5399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423203 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:37.436224 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423205 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423208 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423210 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423213 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423216 5399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423218 5399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423221 5399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423223 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423228 5399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423231 5399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423233 5399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423236 5399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423238 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423241 5399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423243 5399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423246 5399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423250 5399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423253 5399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423255 5399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:37.436748 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423258 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423260 5399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423263 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423265 5399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423268 5399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423283 5399 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423288 5399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423291 5399 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423295 5399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423298 5399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423300 5399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423303 5399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423306 5399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423309 5399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423312 5399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423315 5399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423317 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423320 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423322 5399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:37.437199 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423325 5399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423328 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423331 5399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423334 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423337 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423339 5399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423342 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423345 5399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423347 5399 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423350 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423352 5399 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423357 5399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423361 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423365 5399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423368 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423371 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423373 5399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423376 5399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423378 5399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423381 5399 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:37.437675 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423383 5399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423387 5399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423390 5399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423392 5399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423395 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423398 5399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.423400 5399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.426896 5399 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.426910 5399 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.426927 5399 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.426932 5399 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427101 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427526 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427532 5399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427547 5399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427552 5399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427556 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427561 5399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427565 5399 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427570 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:37.438146 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427575 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427579 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427584 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427590 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427594 5399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427599 5399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427603 5399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427613 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427617 5399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427622 5399 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427627 5399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427631 5399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427636 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427640 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427645 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427649 5399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427653 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427658 5399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427662 5399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427671 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:37.438644 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427676 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427680 5399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427684 5399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427689 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427693 5399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427697 5399 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427702 5399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427707 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427711 5399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427715 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427719 5399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427724 5399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427733 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427737 5399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427742 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427746 5399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427751 5399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427755 5399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427759 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:37.439137 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427763 5399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427768 5399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427772 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427776 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427780 5399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427785 5399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427794 5399 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427798 5399 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427803 5399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427807 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427811 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427816 5399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427820 5399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427824 5399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427828 5399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427832 5399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427837 5399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427841 5399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427851 5399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427855 5399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:37.439641 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427859 5399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427864 5399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427868 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427875 5399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427882 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427888 5399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427894 5399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427899 5399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427903 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427908 5399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427913 5399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427922 5399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427927 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427931 5399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427935 5399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427940 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427944 5399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427949 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:37.440115 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427953 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427957 5399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427962 5399 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427967 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427971 5399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427975 5399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427983 5399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427988 5399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427993 5399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.427997 5399 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428001 5399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428006 5399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428010 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428014 5399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428018 5399 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428023 5399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428027 5399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428033 5399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428044 5399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:37.440563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428049 5399 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.428159 5399 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428625 5399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428634 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428638 5399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428641 5399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428644 5399 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428648 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428651 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428654 5399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428657 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428660 5399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428663 5399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428666 5399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428668 5399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428671 5399 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:37.441013 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428674 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428677 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428679 5399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428682 5399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428685 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428688 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428691 5399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428694 5399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428696 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428699 5399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428702 5399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428704 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428707 5399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428710 5399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428712 5399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428715 5399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428718 5399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428721 5399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428724 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:37.441415 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428727 5399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428729 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428732 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428734 5399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428738 5399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428740 5399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428743 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428746 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428748 5399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428751 5399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428754 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428757 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428760 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428763 5399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428766 5399 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428768 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428771 5399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428773 5399 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428776 5399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428779 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:37.441864 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428781 5399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428784 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428787 5399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428790 5399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428793 5399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428796 5399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428798 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428801 5399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428803 5399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428806 5399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428810 5399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428814 5399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428817 5399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428821 5399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428823 5399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428826 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428829 5399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428832 5399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428834 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:37.442537 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428837 5399 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428840 5399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428842 5399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428845 5399 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428847 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428850 5399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428853 5399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428855 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428858 5399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428860 5399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428863 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428866 5399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428869 5399 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428871 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428874 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428877 5399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428880 5399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428883 5399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428886 5399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:37.442972 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428889 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428893 5399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428895 5399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428898 5399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428900 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428903 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428906 5399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428908 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428911 5399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428914 5399 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428916 5399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428919 5399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428921 5399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428924 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428927 5399 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.428951 5399 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:52:37.443434 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428986 5399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428991 5399 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428994 5399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.428997 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429000 5399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429003 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429006 5399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429009 5399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429012 5399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429014 5399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429017 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429020 5399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429022 5399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429025 5399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429029 5399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429032 5399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429035 5399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429038 5399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429041 5399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:52:37.443807 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429044 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429046 5399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429051 5399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429054 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429057 5399 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429059 5399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429062 5399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429065 5399 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429067 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429070 5399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429072 5399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429075 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429077 5399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429080 5399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429082 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429085 5399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429088 5399 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429091 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429093 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429096 5399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:52:37.444250 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429099 5399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429102 5399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429106 5399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429110 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429112 5399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429115 5399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429118 5399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429120 5399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429123 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429125 5399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429128 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429130 5399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429133 5399 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429135 5399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429139 5399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429142 5399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429145 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429147 5399 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429150 5399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429152 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:52:37.444740 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429155 5399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429157 5399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429160 5399 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429163 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429166 5399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429168 5399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429171 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429174 5399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429177 5399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429180 5399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429182 5399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429185 5399 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429188 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429190 5399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429193 5399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429195 5399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429198 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429200 5399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429203 5399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429205 5399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:52:37.445559 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429208 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429211 5399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429213 5399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429216 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429218 5399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429221 5399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429225 5399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429228 5399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429231 5399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429234 5399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429237 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429239 5399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429242 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429244 5399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429247 5399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429250 5399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429252 5399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429255 5399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429258 5399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:52:37.446377 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429260 5399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429263 5399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429266 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429268 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429291 5399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429295 5399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429297 5399 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:52:37.429300 5399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.429359 5399 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.430294 5399 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.430351 5399 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.430538 5399 server.go:1014] "Starting client certificate rotation" Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.430682 5399 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.430739 5399 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:46:25 +0000 UTC" deadline="2028-07-17 08:30:49.161193904 +0000 UTC" Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.430776 5399 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14940h38m11.730419822s" Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.430996 5399 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:52:37.447183 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.432012 5399 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:52:37.447853 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.433362 5399 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:52:37.447853 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.435207 5399 log.go:25] "Validated CRI v1 image API" Nov 02 19:52:37.447853 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.435929 5399 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:52:37.447853 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.437833 5399 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 def1b433-09c4-4b28-bfd7-bcc798fdae81:/dev/nvme0n1p3 fd7926d3-74c9-4adb-8aab-099fd39106b4:/dev/nvme0n1p4] Nov 02 19:52:37.447989 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.437852 5399 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04212e999252863e939f0a718697e75636fa07bc8c39f1b4c2c9181da3afcd97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04212e999252863e939f0a718697e75636fa07bc8c39f1b4c2c9181da3afcd97/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0786d1a3cabedf3c927e808edce92df0ddb6e74539c11dc41f10b99b669268f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0786d1a3cabedf3c927e808edce92df0ddb6e74539c11dc41f10b99b669268f6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c4cb6289638aa3a12a91a97de8d4b333376eb68b0eb0acbcd8dcfb9f15cdd3b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c4cb6289638aa3a12a91a97de8d4b333376eb68b0eb0acbcd8dcfb9f15cdd3b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ff7077a7fe69feb99960b9c6b29819245a604dfe3769ad309b348b089f86664/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ff7077a7fe69feb99960b9c6b29819245a604dfe3769ad309b348b089f86664/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26e67e5f5480c8e90868081c6f4fc876dd75f50e375542abc6d78ced2ffa2828/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26e67e5f5480c8e90868081c6f4fc876dd75f50e375542abc6d78ced2ffa2828/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ff900718fa6f214f64b5aadb7e4ece46156eb00d770996e5aa287a04ffc9ba1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ff900718fa6f214f64b5aadb7e4ece46156eb00d770996e5aa287a04ffc9ba1/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31155f41c7c63eab90ba61785de776eb7e706838aad039e6c0f93846777f08b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31155f41c7c63eab90ba61785de776eb7e706838aad039e6c0f93846777f08b9/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c390dfedefcfa015389b92b41df59eadbb0341df41d633456a2746c4b7c6c90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c390dfedefcfa015389b92b41df59eadbb0341df41d633456a2746c4b7c6c90/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fd3a04e4b6785a304f94c2ec8aab79af236b59e07d73f10b441dff3fdc0c28d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fd3a04e4b6785a304f94c2ec8aab79af236b59e07d73f10b441dff3fdc0c28d/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f6564f9212779fbfcdac37d7ba506c3720115a643f6c0f7cbf51749d28b6e5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f6564f9212779fbfcdac37d7ba506c3720115a643f6c0f7cbf51749d28b6e5e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b4e7d5968a48e4328cc58f12cbcf6796fd18e4cbfd314d6894474b1dca4a64f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b4e7d5968a48e4328cc58f12cbcf6796fd18e4cbfd314d6894474b1dca4a64f/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7a02aa8ada6156560906284b23afa0a0b77373d07eeb8b1146adf633865b31f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7a02aa8ada6156560906284b23afa0a0b77373d07eeb8b1146adf633865b31f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8ca966ae355cc3d44fc2d72a49ecb5a6e94a29133fc59fef18ffc8150c125c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8ca966ae355cc3d44fc2d72a49ecb5a6e94a29133fc59fef18ffc8150c125c0/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8d0b395b02a2125f88670ba84c7550de9ea2f3e14bdb66b6ef37b54da72864d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8d0b395b02a2125f88670ba84c7550de9ea2f3e14bdb66b6ef37b54da72864d/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3e47f32cb33b982fac5a898072a4791f9a1b54c4b50207a5c7587844802885c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3e47f32cb33b982fac5a898072a4791f9a1b54c4b50207a5c7587844802885c/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf9c00cc60cedf8e413bbcf47d6a2dcd19eafc60b48025d55bdd5a4f8cbc6ca3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf9c00cc60cedf8e413bbcf47d6a2dcd19eafc60b48025d55bdd5a4f8cbc6ca3/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d358bf5747cf075a43e90306382ce0c282d51d4d5aefd7c1ccc42b7256f5e3c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d358bf5747cf075a43e90306382ce0c282d51d4d5aefd7c1ccc42b7256f5e3c1/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dada2124e1ccdfdbbede798a85d63a3031ded078504441eab2b3c8b52f94fbd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dada2124e1ccdfdbbede798a85d63a3031ded078504441eab2b3c8b52f94fbd0/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e188059c884916b475a8da5131e210badc00b011f36195e3b4313c88ad7f9532/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e188059c884916b475a8da5131e210badc00b011f36195e3b4313c88ad7f9532/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2be4ef9fd6b4e93ddbb046eed8ef46af779711960a8902397a7dcda71309f9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2be4ef9fd6b4e93ddbb046eed8ef46af779711960a8902397a7dcda71309f9f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/002bc527-a392-444a-85a5-2dbe005559f4/volumes/kubernetes.io~projected/kube-api-access-6gbmv:{mountpoint:/var/lib/kubelet/pods/002bc527-a392-444a-85a5-2dbe005559f4/volumes/kubernetes.io~projected/kube-api-access-6gbmv major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/002bc527-a392-444a-85a5-2dbe005559f4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/002bc527-a392-444a-85a5-2dbe005559f4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/002bc527-a392-444a-85a5-2dbe005559f4/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/002bc527-a392-444a-85a5-2dbe005559f4/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/040d6f5e-129c-4665-ad54-957415093fa3/volumes/kubernetes.io~projected/kube-api-access-psrcq:{mountpoint:/var/lib/kubelet/pods/040d6f5e-129c-4665-ad54-957415093fa3/volumes/kubernetes.io~projected/kube-api-access-psrcq major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0/volumes/kubernetes.io~projected/kube-api-access-xtpvg:{mountpoint:/var/lib/kubelet/pods/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0/volumes/kubernetes.io~projected/kube-api-access-xtpvg major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36ec01c3-dcbe-4caf-baff-dcade7691e2e/volumes/kubernetes.io~projected/kube-api-access-nhzfq:{mountpoint:/var/lib/kubelet/pods/36ec01c3-dcbe-4caf-baff-dcade7691e2e/volumes/kubernetes.io~projected/kube-api-access-nhzfq major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36ec01c3-dcbe-4caf-baff-dcade7691e2e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/36ec01c3-dcbe-4caf-baff-dcade7691e2e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41e23004-aede-47df-a301-b880f169a803/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/41e23004-aede-47df-a301-b880f169a803/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41e23004-aede-47df-a301-b880f169a803/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/41e23004-aede-47df-a301-b880f169a803/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41e23004-aede-47df-a301-b880f169a803/volumes/kubernetes.io~projected/kube-api-access-sng9v:{mountpoint:/var/lib/kubelet/pods/41e23004-aede-47df-a301-b880f169a803/volumes/kubernetes.io~projected/kube-api-access-sng9v major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c6469de-dc6b-405f-b14f-cf765fd9d06f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4c6469de-dc6b-405f-b14f-cf765fd9d06f/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68/volumes/kubernetes.io~projected/kube-api-access-jkn9t:{mountpoint:/var/lib/kubelet/pods/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68/volumes/kubernetes.io~projected/kube-api-access-jkn9t major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6048b8aa-9e93-4214-8b5a-ed6663b632b5/volumes/kubernetes.io~projected/kube-api-access-l9jvj:{mountpoint:/var/lib/kubelet/pods/6048b8aa-9e93-4214-8b5a-ed6663b632b5/volumes/kubernetes.io~projected/kube-api-access-l9jvj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71685c84-43a3-4272-8d8b-d6c623f8f592/volumes/kubernetes.io~projected/kube-api-access-9s2f7:{mountpoint:/var/lib/kubelet/pods/71685c84-43a3-4272-8d8b-d6c623f8f592/volumes/kubernetes.io~projected/kube-api-access-9s2f7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71685c84-43a3-4272-8d8b-d6c623f8f592/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/71685c84-43a3-4272-8d8b-d6c623f8f592/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volume-subpaths/web-config/config-reloader/2 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volume-subpaths/web-config/prometheus/15 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~empty-dir/config-out major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~projected/kube-api-access-zjk47:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~projected/kube-api-access-zjk47 major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~projected/tls-assets major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/config major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/web-config major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76aa7437-fa4a-42a6-90d5-9043d9b8a71d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/76aa7437-fa4a-42a6-90d5-9043d9b8a71d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2/volumes/kubernetes.io~projected/kube-api-access-plp8r:{mountpoint:/var/lib/kubelet/pods/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2/volumes/kubernetes.io~projected/kube-api-access-plp8r major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/volumes/kubernetes.io~projected/kube-api-access-pqvgg:{mountpoint:/var/lib/kubelet/pods/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/volumes/kubernetes.io~projected/kube-api-access-pqvgg major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/web-config/alertmanager/9 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/web-config/config-reloader/6 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~empty-dir/config-out major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~projected/kube-api-access-vbxnx:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~projected/kube-api-access-vbxnx major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~projected/tls-assets major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/config-volume major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/web-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1/volumes/kubernetes.io~projected/kube-api-access-6wfgp:{mountpoint:/var/lib/kubelet/pods/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1/volumes/kubernetes.io~projected/kube-api-access-6wfgp major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae8ba755-ec98-496c-a59a-4f4625fea774/volumes/kubernetes.io~projected/kube-api-access-5m6hm:{mountpoint:/var/lib/kubelet/pods/ae8ba755-ec98-496c-a59a-4f4625fea774/volumes/kubernetes.io~projected/kube-api-access-5m6hm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b39d4538-dc20-4f6e-8170-4f5c0327cf69/volumes/kubernetes.io~projected/kube-api-access-rcg4w:{mountpoint:/var/lib/kubelet/pods/b39d4538-dc20-4f6e-8170-4f5c0327cf69/volumes/kubernetes.io~projected/kube-api-access-rcg4w major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd701aaf-13ea-4699-9c01-5167947a614a/volumes/kubernetes.io~projected/kube-api-access-w9z9n:{mountpoint:/var/lib/kubelet/pods/cd701aaf-13ea-4699-9c01-5167947a614a/volumes/kubernetes.io~projected/kube-api-access-w9z9n major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd701aaf-13ea-4699-9c01-5167947a614a/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/cd701aaf-13ea-4699-9c01-5167947a614a/volumes/kubernetes.io~secret/metrics-certs major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f81e0175-b8d3-4348-bc49-d9e00531643f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f81e0175-b8d3-4348-bc49-d9e00531643f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f81e0175-b8d3-4348-bc49-d9e00531643f/volumes/kubernetes.io~projected/kube-api-access-hk7j2:{mountpoint:/var/lib/kubelet/pods/f81e0175-b8d3-4348-bc49-d9e00531643f/volumes/kubernetes.io~projected/kube-api-access-hk7j2 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f81e0175-b8d3-4348-bc49-d9e00531643f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f81e0175-b8d3-4348-bc49-d9e00531643f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7925bc98144c23f5164a14be0ce6adefd5ec3c1c7015219b5cce4ddd46f4ab09/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a7a31c91bee2f742495218fb89b0132603e527c2d06215c822867084a840ad0f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/908e7d977ed18b74de7517c33f3b851bd4a07d45c489fe6be396882c4b6e470d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/209484b7f17e06af14ef19c54c0c3190392dc96c5d6a90a95bb392ad0ee2911e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/17cf94e919477a6fb3f7e3b830d408409af92e4c2445db499d9847e76eca8c3e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5674b360ebf665c624afb9bd3fdee3d16b2ef1195ea0a256e3b3686d9c9c9a9a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/727f15ec1bce078d00b9d7b724d4810984026d9f9222c339559ef15492c5fb0b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/88b956a61809a2171c951ae3c5795167770dbb8f713bbc597b0b72d06ff8a37e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3d286150bfab0bc1aeabc5987854e0fc25f94046be1ccd8916ce571a1fa235a0/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e37e4cd1d94519eefe08d5e66efb35c9cf9a7b17e1321974490feb9c0d08c4ab/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/fa9b83fbe2cdcb6e0f4c7b0337fd5f32f98ffd49a71627f21659c8725fc60c84/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/205bed8df588a6b53f6c3f6c306b7ef50dc73c1634f21e7619c9c6cfd4230e6c/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/44273e413c93f6cea6daf715d93f9a210350daa605d23b39a5de7af6db04ab01/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3dbc84a530921c33036d5de0c00b029ace72f8fef1bd721e6c166b3094e291df/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/331d7239f05497b85480b56c96defbf3e62a65dd5c4e9cb8fcee4325c7be2bac/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2fc30a86a4a86a75abcd87690b59e2ba9ea0fd129ffb4b5062a6a4700a319c9f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5ca0eb97e8ce6736d31a80886b5f8975262ec0de86cd04114871c6dc2bccc784/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e50463a26db8a1a1fe371d852082451adcc2debd77cbf3f79532dbb846ed4043/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/13f3b77e4a1f606f7f0dafea3ef5bb197d6eb6f79d7f07e259ac62592bb8643f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cef3d6a28109698f8d4b051b9134add814f6aab9e0a87a8904c1fc0c8701aacb/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b425817cdfe50f5fe41f45f53dc90b1fd9dca9edad7039360328744a7a869fca/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/5a89936f3476f87fb159a38d37a17d4b25895ac9a35626250c5bc2758fb0b026/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/18fdf9242f8ed9f27a1054b5af3c6a803c4cef0ba4c38c13445c4a545bd63f72/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/5784fda8325361ee07a14ae7e05f8bb6810e50c0dcb0e2eec351a828901d8609/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/da80b6c0ed712adf23727fd25ccccc9ad57aedfbfd597fee5b9c15087a3263de/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/55db2b812f1eb531b46e3387d1ee5aad549b7a2bd35f3bf08471dca644c4151b/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/437d877d99009dd26aa1ed1993c995793151485fc11c6dce70f1ca13d786cccb/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/f2164b00868eb2640cc2d3e90aa4bae2af8fba47d9d736e9ff6ffd2c14cb3077/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/32b2fc6fd7079997488b8e845f28b9bde56b7557eafaff22d11c75aaf02c57c7/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/79af3c222d356f5f4bb13d74b185355dfff351cdff97945e9d65aac4c12bd809/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/4de2c4d48351cc047405800a65f6845a73c9c0a499a422f5a5f5cd7b90adb101/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/e650cd873074bf483a48a2b91ecf3341496c4d1e7e6df3b095e79c608b1a5c6d/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/e8ea89735f7975c2e44e4107a8fc6500b46e478c007bfbc53131cbe7f91ade2c/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/0ff2ef4da45e67e8030abff6294d95c8d6e6f555c67280082731231765a5f9ff/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/6a80a9984b665b6d2cd4dc05eb95a4190c905ecaa44162db0fe90cf4ae140d71/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/d79872b005a75c4286e4a3781febc34f6a69a6c9b4e07549c486c815e0ca7a75/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/cd70c7180cdfb34166aa640bb02e53466478d2f9e2b4ee64323aa9833c883295/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/f89024931ec2253c0d22f949886bdce17dbd9a5943ca462af3ad4eaeb7c731d2/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/44959d2d6efbb8fae7ba20824ac7eab6910ea3556efd033f6914fabd0fb65963/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/aacbef45dd27b466558b6ab1a202ad6ea226ef78d36bbd79a4a9b63496b91f15/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/786f792db2ebe5358d4a82bb23856739bb8c57c6e2a75e54a2417c0d85ee62e3/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/43e410bd2a321dee8a7dea792de7aef4c7f707255719809d598c8686e67f74b0/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/f4014f4ac7db593fd5a69be6714f3694b0e098b34e42affaf8ac271bf0ffe2c7/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/93c8fbf6e9fa859a0010d7e216564815c894ac338aed50c6d79a2bea03a6b441/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/bb46ce6cdb14f9fcc3b30bd2f2e419221e41b61c58deb22ed783126919cc12bb/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/673c7694abc9d4637b086f6e05919e37e49b1f50e0cf41a96e7b04dc2c59c3cf/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/8ca946449fd1da615c634514d4f49df22b9f0136b5bb74522436866f8d8f6572/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/f5d6e0cd47a1fe35ba70c0b2f861685b41fc2294a5a9031830f663534f20df96/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/1fc3619a579966901e3bff0a50332fa62bdce590656938c9972e6e63f1187813/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/9099804ae84149e473db63f8901db9ce1e8d19831c7f65aa9564fd6394f1f15d/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/6e94ca916860f93f42f8eaedd9d445bd584c780754b736bdbbeb512836c7d8b4/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/b29a3caed024e2b5a6d3a63f408078fb210a1d148e7ad66bd55b2728a61f61fc/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/ba4c093ac96508a6940ec6d2e7261d34677b459165b559b67dee5d56e0c6e6ea/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/aab6a1bd9119875c29113daf15007282e2d7b4719031ab4345455e7041d235bd/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/79f5ffbca3a11c7f6ebf1b6b8d6e3398c534b921239276e1c9655117f51f8904/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/440d400b5a087b287ce47cff555645508d9d76c1e542b62fb0e47336652a7575/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/30704dc40385dc5730a56d2ceb948b8265ab004fd9f23ba9976df94da2d80353/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e2d90e4b3d46b4a84bbb582f8fa20866453427687ee31310f9b01f5bd8010990/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/21025193b05360070136177ea98a655ddb827ce17d17b0da6b05de6a95d2250a/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/44d7a2260c745b68e7d01a6a7bac282275aeda9c8292b3aff6fe363d8823c2c1/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/41b22a380bb4b908029acc6304346c22e69988a51bf00e62febcf69018934d36/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/af829a56c2cdc183d4f0b96b0e1e8b9b68c267bdf6126fc75a53d67524c1a1ba/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/88208485fddf7c14ee4c7a2bf76e6b812a474b516c9a415b59b03a18e20dc29e/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.446615 5399 manager.go:217] Machine: {Timestamp:2026-11-02 19:52:37.445573325 +0000 UTC m=+0.056225409 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107742 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20cac3cf9745bcfeaf4e7f2f2d4771 SystemUUID:ec20cac3-cf97-45bc-feaf-4e7f2f2d4771 BootID:2a5aef47-648f-4978-909a-980959a584e2 Filesystems:[{Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b4e7d5968a48e4328cc58f12cbcf6796fd18e4cbfd314d6894474b1dca4a64f/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~projected/kube-api-access-vbxnx DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f81e0175-b8d3-4348-bc49-d9e00531643f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/cd701aaf-13ea-4699-9c01-5167947a614a/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:326 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:347 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68/volumes/kubernetes.io~projected/kube-api-access-jkn9t DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e188059c884916b475a8da5131e210badc00b011f36195e3b4313c88ad7f9532/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4c6469de-dc6b-405f-b14f-cf765fd9d06f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:348 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36ec01c3-dcbe-4caf-baff-dcade7691e2e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c390dfedefcfa015389b92b41df59eadbb0341df41d633456a2746c4b7c6c90/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41e23004-aede-47df-a301-b880f169a803/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:276 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/040d6f5e-129c-4665-ad54-957415093fa3/volumes/kubernetes.io~projected/kube-api-access-psrcq DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3e47f32cb33b982fac5a898072a4791f9a1b54c4b50207a5c7587844802885c/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dada2124e1ccdfdbbede798a85d63a3031ded078504441eab2b3c8b52f94fbd0/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26e67e5f5480c8e90868081c6f4fc876dd75f50e375542abc6d78ced2ffa2828/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41e23004-aede-47df-a301-b880f169a803/volumes/kubernetes.io~projected/kube-api-access-sng9v DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/f81e0175-b8d3-4348-bc49-d9e00531643f/volumes/kubernetes.io~projected/kube-api-access-hk7j2 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c4cb6289638aa3a12a91a97de8d4b333376eb68b0eb0acbcd8dcfb9f15cdd3b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:275 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:349 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31155f41c7c63eab90ba61785de776eb7e706838aad039e6c0f93846777f08b9/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/002bc527-a392-444a-85a5-2dbe005559f4/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd701aaf-13ea-4699-9c01-5167947a614a/volumes/kubernetes.io~projected/kube-api-access-w9z9n DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d358bf5747cf075a43e90306382ce0c282d51d4d5aefd7c1ccc42b7256f5e3c1/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71685c84-43a3-4272-8d8b-d6c623f8f592/volumes/kubernetes.io~projected/kube-api-access-9s2f7 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0786d1a3cabedf3c927e808edce92df0ddb6e74539c11dc41f10b99b669268f6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:346 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf9c00cc60cedf8e413bbcf47d6a2dcd19eafc60b48025d55bdd5a4f8cbc6ca3/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/002bc527-a392-444a-85a5-2dbe005559f4/volumes/kubernetes.io~projected/kube-api-access-6gbmv DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f81e0175-b8d3-4348-bc49-d9e00531643f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76aa7437-fa4a-42a6-90d5-9043d9b8a71d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71685c84-43a3-4272-8d8b-d6c623f8f592/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fd3a04e4b6785a304f94c2ec8aab79af236b59e07d73f10b441dff3fdc0c28d/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8d0b395b02a2125f88670ba84c7550de9ea2f3e14bdb66b6ef37b54da72864d/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41e23004-aede-47df-a301-b880f169a803/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f6564f9212779fbfcdac37d7ba506c3720115a643f6c0f7cbf51749d28b6e5e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2be4ef9fd6b4e93ddbb046eed8ef46af779711960a8902397a7dcda71309f9f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1/volumes/kubernetes.io~projected/kube-api-access-6wfgp DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b39d4538-dc20-4f6e-8170-4f5c0327cf69/volumes/kubernetes.io~projected/kube-api-access-rcg4w DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6048b8aa-9e93-4214-8b5a-ed6663b632b5/volumes/kubernetes.io~projected/kube-api-access-l9jvj DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:281 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ff7077a7fe69feb99960b9c6b29819245a604dfe3769ad309b348b089f86664/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/002bc527-a392-444a-85a5-2dbe005559f4/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:344 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7a02aa8ada6156560906284b23afa0a0b77373d07eeb8b1146adf633865b31f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/36ec01c3-dcbe-4caf-baff-dcade7691e2e/volumes/kubernetes.io~projected/kube-api-access-nhzfq DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:342 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04212e999252863e939f0a718697e75636fa07bc8c39f1b4c2c9181da3afcd97/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae8ba755-ec98-496c-a59a-4f4625fea774/volumes/kubernetes.io~projected/kube-api-access-5m6hm DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes/kubernetes.io~projected/kube-api-access-zjk47 DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0/volumes/kubernetes.io~projected/kube-api-access-xtpvg DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2/volumes/kubernetes.io~projected/kube-api-access-plp8r DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/volumes/kubernetes.io~projected/kube-api-access-pqvgg DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8ca966ae355cc3d44fc2d72a49ecb5a6e94a29133fc59fef18ffc8150c125c0/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ff900718fa6f214f64b5aadb7e4ece46156eb00d770996e5aa287a04ffc9ba1/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1ff7077a7fe69fe MacAddress:e6:29:77:8a:b1:de Speed:10000 Mtu:8901} {Name:26e67e5f5480c8e MacAddress:46:b5:f2:77:d8:7b Speed:10000 Mtu:8901} {Name:4c390dfedefcfa0 MacAddress:ca:0f:92:8b:59:60 Speed:10000 Mtu:8901} {Name:4fd3a04e4b6785a MacAddress:7a:b6:f8:3b:8b:4a Speed:10000 Mtu:8901} {Name:a8ca966ae355cc3 MacAddress:fa:af:8a:14:f1:c0 Speed:10000 Mtu:8901} {Name:b3e47f32cb33b98 MacAddress:5e:27:47:f4:b8:2a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c5:e5:bd:a9 Speed:0 Mtu:9001} {Name:br-int MacAddress:46:39:71:44:aa:d5 Speed:0 Mtu:8901} {Name:cf9c00cc60cedf8 MacAddress:7e:8f:00:c3:51:3b Speed:10000 Mtu:8901} {Name:dada2124e1ccdfd MacAddress:46:e0:1f:4a:f5:47 Speed:10000 Mtu:8901} {Name:e188059c884916b MacAddress:3a:d5:a2:58:bf:81 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c5:e5:bd:a9 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:06:03:05:8a:46:4a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:d3:58:a0:29:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.447256 5399 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.447328 5399 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.447612 5399 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.447639 5399 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-111.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.447757 5399 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.447765 5399 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.447776 5399 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.447789 5399 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.447863 5399 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.448019 5399 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.448123 5399 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.448156 5399 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.448165 5399 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.448174 5399 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.448183 5399 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:52:37.448213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.448190 5399 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:52:37.448996 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.448787 5399 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:52:37.448996 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.448796 5399 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:52:37.449241 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.449213 5399 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:52:37.460810 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.460790 5399 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:52:37.460810 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.460805 5399 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:52:37.460935 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.460903 5399 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 19:52:37.461103 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461091 5399 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:52:37.461138 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461121 5399 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:52:37.461219 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461210 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:52:37.461266 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461225 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:52:37.461266 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461231 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:52:37.461266 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461237 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:52:37.461266 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461242 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:52:37.461266 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461248 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:52:37.461266 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461254 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:52:37.461266 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461260 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:52:37.461266 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461267 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:52:37.461592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461291 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:52:37.461592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461302 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:52:37.461592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461311 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:52:37.461592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461339 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:52:37.461592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.461345 5399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:52:37.464804 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.464792 5399 server.go:1263] "Started kubelet" Nov 02 19:52:37.465473 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.464870 5399 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:52:37.465473 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.464900 5399 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:52:37.465473 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.465013 5399 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:52:37.465365 ip-10-0-130-111 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:52:37.466531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.465500 5399 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:52:37.466890 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.466798 5399 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:52:37.470553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.469803 5399 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:52:37.470553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.469995 5399 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:52:37.470553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.470011 5399 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:52:37.470553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.469870 5399 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:52:37.470553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.470077 5399 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:46:27 +0000 UTC" deadline="2028-05-05 02:14:13.461222855 +0000 UTC" Nov 02 19:52:37.470553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.470110 5399 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:52:37.470553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.470115 5399 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13182h21m35.991111703s" Nov 02 19:52:37.470553 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:37.470174 5399 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:52:37.472653 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.471836 5399 factory.go:153] Registering CRI-O factory Nov 02 19:52:37.472653 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.471859 5399 factory.go:223] Registration of the crio container factory successfully Nov 02 19:52:37.472653 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.471929 5399 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:52:37.472653 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.471941 5399 factory.go:55] Registering systemd factory Nov 02 19:52:37.472653 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.471948 5399 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:52:37.472653 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.471983 5399 factory.go:103] Registering Raw factory Nov 02 19:52:37.472653 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.471996 5399 manager.go:1199] Started watching for new ooms in manager Nov 02 19:52:37.473045 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.472662 5399 manager.go:319] Starting recovery of all containers Nov 02 19:52:37.473820 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473782 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f81e0175-b8d3-4348-bc49-d9e00531643f" volumeName="kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-config" seLinuxMountContext="" Nov 02 19:52:37.473912 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473829 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002bc527-a392-444a-85a5-2dbe005559f4" volumeName="kubernetes.io/projected/002bc527-a392-444a-85a5-2dbe005559f4-kube-api-access-6gbmv" seLinuxMountContext="" Nov 02 19:52:37.473912 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473844 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 19:52:37.473912 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473855 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2" volumeName="kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cni-binary-copy" seLinuxMountContext="" Nov 02 19:52:37.473912 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473867 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f81e0175-b8d3-4348-bc49-d9e00531643f" volumeName="kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-env-overrides" seLinuxMountContext="" Nov 02 19:52:37.473912 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473879 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002bc527-a392-444a-85a5-2dbe005559f4" volumeName="kubernetes.io/empty-dir/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-textfile" seLinuxMountContext="" Nov 02 19:52:37.473912 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473891 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e35ec0a-ad15-48bc-ac59-e08c8dec75a0" volumeName="kubernetes.io/empty-dir/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-tmp-dir" seLinuxMountContext="" Nov 02 19:52:37.473912 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473902 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473917 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473929 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473940 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002bc527-a392-444a-85a5-2dbe005559f4" volumeName="kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473951 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6048b8aa-9e93-4214-8b5a-ed6663b632b5" volumeName="kubernetes.io/projected/6048b8aa-9e93-4214-8b5a-ed6663b632b5-kube-api-access-l9jvj" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473964 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98" volumeName="kubernetes.io/projected/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-kube-api-access-pqvgg" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473978 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-metrics-client-ca" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.473992 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-kube-api-access-vbxnx" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474006 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39d4538-dc20-4f6e-8170-4f5c0327cf69" volumeName="kubernetes.io/configmap/b39d4538-dc20-4f6e-8170-4f5c0327cf69-serviceca" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474019 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f81e0175-b8d3-4348-bc49-d9e00531643f" volumeName="kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-script-lib" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474031 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474042 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="040d6f5e-129c-4665-ad54-957415093fa3" volumeName="kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474055 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6048b8aa-9e93-4214-8b5a-ed6663b632b5" volumeName="kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474088 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474100 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474111 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-main-db" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474123 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-cluster-tls-config" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474135 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae8ba755-ec98-496c-a59a-4f4625fea774" volumeName="kubernetes.io/projected/ae8ba755-ec98-496c-a59a-4f4625fea774-kube-api-access-5m6hm" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474148 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474160 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-web-config" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474172 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1" volumeName="kubernetes.io/empty-dir/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-tmp-dir" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474189 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1" volumeName="kubernetes.io/projected/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-kube-api-access-6wfgp" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474203 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f81e0175-b8d3-4348-bc49-d9e00531643f" volumeName="kubernetes.io/projected/f81e0175-b8d3-4348-bc49-d9e00531643f-kube-api-access-hk7j2" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474228 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e71e3e7-20f2-4dbd-bd77-1debbbdacf68" volumeName="kubernetes.io/projected/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-kube-api-access-jkn9t" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474243 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6048b8aa-9e93-4214-8b5a-ed6663b632b5" volumeName="kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-binary-copy" seLinuxMountContext="" Nov 02 19:52:37.474260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474254 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-tls-assets" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474308 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1" volumeName="kubernetes.io/configmap/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-config-volume" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474335 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f81e0175-b8d3-4348-bc49-d9e00531643f" volumeName="kubernetes.io/secret/f81e0175-b8d3-4348-bc49-d9e00531643f-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474348 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71685c84-43a3-4272-8d8b-d6c623f8f592" volumeName="kubernetes.io/projected/71685c84-43a3-4272-8d8b-d6c623f8f592-kube-api-access-9s2f7" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474360 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-kube-api-access-zjk47" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474373 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-web-config" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474385 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002bc527-a392-444a-85a5-2dbe005559f4" volumeName="kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474399 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6048b8aa-9e93-4214-8b5a-ed6663b632b5" volumeName="kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474413 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474425 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-out" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474437 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474451 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1" volumeName="kubernetes.io/secret/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-metrics-tls" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474463 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ec01c3-dcbe-4caf-baff-dcade7691e2e" volumeName="kubernetes.io/configmap/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474476 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ec01c3-dcbe-4caf-baff-dcade7691e2e" volumeName="kubernetes.io/projected/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-api-access-nhzfq" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474488 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-db" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474500 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-volume" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474512 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474525 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-tls-assets" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474537 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474550 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474562 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41e23004-aede-47df-a301-b880f169a803" volumeName="kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-tmp" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474574 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c6469de-dc6b-405f-b14f-cf765fd9d06f" volumeName="kubernetes.io/secret/4c6469de-dc6b-405f-b14f-cf765fd9d06f-agent-certs" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474587 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e71e3e7-20f2-4dbd-bd77-1debbbdacf68" volumeName="kubernetes.io/configmap/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-iptables-alerter-script" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474601 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config-out" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474622 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474634 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41e23004-aede-47df-a301-b880f169a803" volumeName="kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-etc-tuned" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474647 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41e23004-aede-47df-a301-b880f169a803" volumeName="kubernetes.io/projected/41e23004-aede-47df-a301-b880f169a803-kube-api-access-sng9v" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474658 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c6469de-dc6b-405f-b14f-cf765fd9d06f" volumeName="kubernetes.io/configmap/4c6469de-dc6b-405f-b14f-cf765fd9d06f-konnectivity-ca" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474669 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474683 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2" volumeName="kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-daemon-config" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474696 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98" volumeName="kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-tls" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474708 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98" volumeName="kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474719 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94d53725-3f49-4c81-b68a-1a60a69a2fbc" volumeName="kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474731 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002bc527-a392-444a-85a5-2dbe005559f4" volumeName="kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-metrics-client-ca" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474743 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e35ec0a-ad15-48bc-ac59-e08c8dec75a0" volumeName="kubernetes.io/projected/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-kube-api-access-xtpvg" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474756 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71685c84-43a3-4272-8d8b-d6c623f8f592" volumeName="kubernetes.io/secret/71685c84-43a3-4272-8d8b-d6c623f8f592-cert" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474769 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2" volumeName="kubernetes.io/projected/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-kube-api-access-plp8r" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474781 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd701aaf-13ea-4699-9c01-5167947a614a" volumeName="kubernetes.io/projected/cd701aaf-13ea-4699-9c01-5167947a614a-kube-api-access-w9z9n" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474793 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474806 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474818 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76aa7437-fa4a-42a6-90d5-9043d9b8a71d" volumeName="kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474832 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98" volumeName="kubernetes.io/configmap/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-metrics-client-ca" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474844 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b39d4538-dc20-4f6e-8170-4f5c0327cf69" volumeName="kubernetes.io/projected/b39d4538-dc20-4f6e-8170-4f5c0327cf69-kube-api-access-rcg4w" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474856 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd701aaf-13ea-4699-9c01-5167947a614a" volumeName="kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474868 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ec01c3-dcbe-4caf-baff-dcade7691e2e" volumeName="kubernetes.io/secret/36ec01c3-dcbe-4caf-baff-dcade7691e2e-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474903 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="002bc527-a392-444a-85a5-2dbe005559f4" volumeName="kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-tls" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474916 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ec01c3-dcbe-4caf-baff-dcade7691e2e" volumeName="kubernetes.io/empty-dir/36ec01c3-dcbe-4caf-baff-dcade7691e2e-data-volume" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474930 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474941 5399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f8744c-1c79-4c6a-b2fb-c7aeac335710" volumeName="kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-grpc-tls" seLinuxMountContext="" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474953 5399 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:52:37.475469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.474961 5399 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:52:37.498028 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.498001 5399 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:52:37.499576 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.499557 5399 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:52:37.499576 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.499579 5399 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:52:37.499713 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.499601 5399 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:52:37.499713 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:37.499643 5399 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 19:52:37.518423 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.518396 5399 generic.go:356] "Generic (PLEG): container finished" podID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerID="edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded" exitCode=0 Nov 02 19:52:37.520769 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.520745 5399 generic.go:356] "Generic (PLEG): container finished" podID="002bc527-a392-444a-85a5-2dbe005559f4" containerID="0e195e598b6a0f38943e8df31d0589058f4e23dcafb44582fb4ba6b4e4297967" exitCode=0 Nov 02 19:52:37.531262 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.531235 5399 generic.go:356] "Generic (PLEG): container finished" podID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerID="3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c" exitCode=0 Nov 02 19:52:37.537823 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.537801 5399 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="0fe98755723a0c3efbb9115265c1c33e6b9434c73012b87deb48f1a956621b28" exitCode=0 Nov 02 19:52:37.537823 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.537820 5399 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="2b76b5a6fd18e447f80b09bde6b7f8df7623948983d8f11f002d980b2f314255" exitCode=0 Nov 02 19:52:37.537823 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.537826 5399 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="4a05283823901fb0d6df87c0d2acf3972915d81fc91f8f514bf8bdab3857f0ca" exitCode=0 Nov 02 19:52:37.537971 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.537832 5399 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="bdfb4b1e2dd96a5d8576f43a6ab0dcd19f39e01aa87398e7b07e3e0b0fd9d03c" exitCode=0 Nov 02 19:52:37.537971 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.537838 5399 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="dcd309aa3b6e63d71e8db375f5b178296e14a3796e3500f7b070afd49bdca524" exitCode=0 Nov 02 19:52:37.537971 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.537843 5399 generic.go:356] "Generic (PLEG): container finished" podID="6048b8aa-9e93-4214-8b5a-ed6663b632b5" containerID="22050b034c7f573d08559a6d882fcd268715721e89af3c3b14fe281e5eb2f20f" exitCode=0 Nov 02 19:52:37.546451 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.546427 5399 generic.go:356] "Generic (PLEG): container finished" podID="27fa11bb17868be078466e66734ae46b" containerID="5af2354cc68bb91686fd7910e17fabb892d3b2762b44e469435ff9353c28bfe7" exitCode=0 Nov 02 19:52:37.563746 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.563730 5399 manager.go:324] Recovery completed Nov 02 19:52:37.570238 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:37.570221 5399 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:52:37.579980 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.579966 5399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:52:37.582476 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.582457 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:52:37.582536 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.582493 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:52:37.582536 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.582509 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:52:37.585260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585238 5399 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:52:37.585260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585255 5399 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:52:37.585404 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585293 5399 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:52:37.585490 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585473 5399 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 19:52:37.585544 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585491 5399 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 19:52:37.585544 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585525 5399 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 19:52:37.585544 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585535 5399 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 19:52:37.585544 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585543 5399 policy_none.go:50] "Start" Nov 02 19:52:37.585726 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585558 5399 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:52:37.585726 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585577 5399 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:52:37.585828 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585815 5399 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 19:52:37.585886 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585831 5399 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 19:52:37.585886 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.585841 5399 policy_none.go:44] "Start" Nov 02 19:52:37.586290 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.586254 5399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:52:37.587551 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.587533 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:52:37.587631 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.587566 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:52:37.587631 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.587579 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:52:37.590689 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.590665 5399 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:52:37.593446 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.593432 5399 manager.go:341] "Starting Device Plugin manager" Nov 02 19:52:37.593517 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:37.593457 5399 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:52:37.593517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.593467 5399 server.go:87] "Starting device plugin registration server" Nov 02 19:52:37.593711 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.593700 5399 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:52:37.593740 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.593713 5399 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:52:37.593817 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.593796 5399 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:52:37.593942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.593899 5399 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:52:37.594048 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.593953 5399 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:52:37.594048 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.593960 5399 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:52:37.594144 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.593997 5399 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:52:37.593937068Z","UUID":"d8850050-be70-445c-aae9-580afa3f6a1c","Handler":null,"Name":"","Endpoint":""} Nov 02 19:52:37.595382 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:37.595357 5399 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:52:37.595486 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:37.595402 5399 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" Nov 02 19:52:37.595486 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.595421 5399 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:52:37.595486 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.595438 5399 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:52:37.599742 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.599710 5399 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal"] Nov 02 19:52:37.599855 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.599841 5399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:52:37.600741 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.600722 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:52:37.600813 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.600753 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:52:37.600813 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.600766 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:52:37.601053 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.601041 5399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:52:37.601162 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.601150 5399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:52:37.601855 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.601838 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:52:37.601935 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.601864 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:52:37.601935 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.601878 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:52:37.602033 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.601967 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:52:37.602033 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.601991 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:52:37.602033 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.602001 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:52:37.602147 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:37.602036 5399 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" node="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.602358 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.602326 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" event={"ID":"ee4b9ca33e3caae4f62b24687213e3f8","Type":"ContainerStarted","Data":"d8cebce6667ad01c532a203d5a566b3be2a0a3abcda97df1a5c340c839eaf7ce"} Nov 02 19:52:37.602397 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.602366 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" event={"ID":"ee4b9ca33e3caae4f62b24687213e3f8","Type":"ContainerStarted","Data":"0786d1a3cabedf3c927e808edce92df0ddb6e74539c11dc41f10b99b669268f6"} Nov 02 19:52:37.602397 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.602373 5399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:52:37.602487 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.602422 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" event={"ID":"27fa11bb17868be078466e66734ae46b","Type":"ContainerStarted","Data":"64e8448abf20e7ae50711c531a4fe87a02a4ff0042f9d972613b91e909b45644"} Nov 02 19:52:37.602487 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.602439 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" event={"ID":"27fa11bb17868be078466e66734ae46b","Type":"ContainerDied","Data":"5af2354cc68bb91686fd7910e17fabb892d3b2762b44e469435ff9353c28bfe7"} Nov 02 19:52:37.602487 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.602451 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" event={"ID":"27fa11bb17868be078466e66734ae46b","Type":"ContainerStarted","Data":"a8d0b395b02a2125f88670ba84c7550de9ea2f3e14bdb66b6ef37b54da72864d"} Nov 02 19:52:37.603150 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.603132 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:52:37.603211 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.603163 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:52:37.603211 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.603175 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:52:37.603332 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:37.603316 5399 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-111.us-west-2.compute.internal\" not found" node="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.654220 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.654197 5399 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:52:37.654394 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.654381 5399 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 19:52:37.654492 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.654477 5399 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:52:37.654556 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.654544 5399 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:52:37.670698 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.670650 5399 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.694852 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.694834 5399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:52:37.696101 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.696077 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:52:37.696206 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.696111 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:52:37.696206 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.696124 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:52:37.696206 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.696182 5399 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.746545 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.746518 5399 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:52:37.746678 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.746629 5399 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:52:37.746732 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:37.746690 5399 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.746732 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.746708 5399 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.755349 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.755316 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.755349 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.755355 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.755442 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.755371 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee4b9ca33e3caae4f62b24687213e3f8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"ee4b9ca33e3caae4f62b24687213e3f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.796041 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.796018 5399 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.796142 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.796079 5399 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.796649 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.796633 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeNotReady" Nov 02 19:52:37.796719 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.796665 5399 setters.go:546] "Node became not ready" node="ip-10-0-130-111.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T19:52:37Z","lastTransitionTime":"2026-11-02T19:52:37Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 19:52:37.826614 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.826591 5399 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:52:37.826705 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:37.826630 5399 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.848654 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.848633 5399 nodeinfomanager.go:471] existing CSINode "ip-10-0-130-111.us-west-2.compute.internal" is owned by different node (oldNodeID="13287dbf-ff47-40f8-8938-7a21ade0fa92", newNodeID=""), cleaning up... Nov 02 19:52:37.855706 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.855687 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.855777 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.855718 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.855777 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.855737 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee4b9ca33e3caae4f62b24687213e3f8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"ee4b9ca33e3caae4f62b24687213e3f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.855845 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.855775 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee4b9ca33e3caae4f62b24687213e3f8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"ee4b9ca33e3caae4f62b24687213e3f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.855845 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.855803 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:37.855845 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:37.855821 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27fa11bb17868be078466e66734ae46b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" (UID: \"27fa11bb17868be078466e66734ae46b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:38.449377 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.449341 5399 apiserver.go:52] "Watching apiserver" Nov 02 19:52:38.545643 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.545606 5399 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:52:38.546216 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.546190 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-zg4fd","kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal","openshift-image-registry/node-ca-rmpzb","openshift-monitoring/node-exporter-x4ttn","openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g","openshift-cluster-node-tuning-operator/tuned-2nprn","openshift-monitoring/prometheus-k8s-0","openshift-dns/dns-default-87h2b","openshift-multus/network-metrics-daemon-7nt8t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w","openshift-dns/node-resolver-nc9rw","openshift-multus/multus-additional-cni-plugins-bn5zl","openshift-multus/multus-gspv7","openshift-ingress-canary/ingress-canary-stttr","openshift-network-operator/iptables-alerter-pvp2h","kube-system/global-pull-secret-syncer-wkg2t","openshift-insights/insights-runtime-extractor-pst9l","kube-system/konnectivity-agent-z28kx","openshift-ovn-kubernetes/ovnkube-node-7m6dl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal","openshift-monitoring/alertmanager-main-0"] Nov 02 19:52:38.552722 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.552692 5399 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:38.552841 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.552738 5399 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:38.571888 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.571863 5399 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:52:38.596117 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.596089 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-shcvr\"" Nov 02 19:52:38.596117 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.596108 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:52:38.596265 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.596158 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vl78h\"" Nov 02 19:52:38.596265 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.596175 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gf4xf\"" Nov 02 19:52:38.596265 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.596224 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dbl8w\"" Nov 02 19:52:38.596834 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.596812 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.596936 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.596865 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.597186 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597173 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:52:38.597186 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597146 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:52:38.597315 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597204 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:52:38.597315 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597211 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.597405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597315 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:52:38.597405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597330 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlz48\"" Nov 02 19:52:38.597405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597348 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.597542 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597422 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.597598 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597564 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.597598 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597577 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.597598 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597584 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.597730 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597587 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4hfb\"" Nov 02 19:52:38.597730 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597638 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f8f9p\"" Nov 02 19:52:38.597730 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597646 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:52:38.597730 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597653 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.597730 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597672 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.597730 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597694 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:52:38.597982 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597774 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:52:38.597982 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597795 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.597982 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597804 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:52:38.597982 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597810 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:52:38.597982 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.597899 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:52:38.598199 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598044 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.598199 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598112 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.598199 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598135 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.598199 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598181 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4w4l2\"" Nov 02 19:52:38.598339 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598267 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:52:38.598339 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598317 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:52:38.598444 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598430 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.598501 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598431 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sk47\"" Nov 02 19:52:38.598501 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598481 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.598567 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598520 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ltmn5\"" Nov 02 19:52:38.598604 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.598573 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qqgmv\"" Nov 02 19:52:38.599042 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.599028 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:52:38.599148 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.599136 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:52:38.599287 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.599256 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:52:38.599287 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.599264 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.599409 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.599396 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.599451 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.599438 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:52:38.599515 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.599502 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d5pq8\"" Nov 02 19:52:38.599544 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.599519 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7qs77\"" Nov 02 19:52:38.599579 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.599566 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8m7zt\"" Nov 02 19:52:38.599610 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.599571 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.648142 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.648125 5399 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:52:38.648214 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:38.648197 5399 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:38.648365 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.648351 5399 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:52:38.648423 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:38.648390 5399 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-111.us-west-2.compute.internal" Nov 02 19:52:38.660172 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660155 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-modprobe-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.660215 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660178 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.660215 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660198 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.660289 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660213 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sng9v\" (UniqueName: \"kubernetes.io/projected/41e23004-aede-47df-a301-b880f169a803-kube-api-access-sng9v\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.660289 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660232 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.660289 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660267 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-multus\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.660377 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660342 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cni-binary-copy\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.660447 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660430 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.660504 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660462 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.660504 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660491 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.660598 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660504 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.660598 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660516 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.660598 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660537 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39d4538-dc20-4f6e-8170-4f5c0327cf69-host\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:52:38.660598 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660562 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-kubernetes\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.660598 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660578 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-tls\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.660598 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660593 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-kubelet\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.660778 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660609 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.660778 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660630 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cni-binary-copy\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.660778 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660637 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-system-cni-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.660778 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660681 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.660778 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660714 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2f7\" (UniqueName: \"kubernetes.io/projected/71685c84-43a3-4272-8d8b-d6c623f8f592-kube-api-access-9s2f7\") pod \"ingress-canary-stttr\" (UID: \"71685c84-43a3-4272-8d8b-d6c623f8f592\") " pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:38.660778 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660731 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-dbus\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:38.660778 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660747 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-env-overrides\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.660778 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660761 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660784 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-k8s-cni-cncf-io\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660809 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-kubelet\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660840 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-conf-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660860 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660875 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-tmp\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660914 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-netns\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660926 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-env-overrides\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660978 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660982 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-tmp\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.660989 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-log-socket\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.661047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661017 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f81e0175-b8d3-4348-bc49-d9e00531643f-ovn-node-metrics-cert\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.661410 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661061 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cnibin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.661410 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661099 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzfq\" (UniqueName: \"kubernetes.io/projected/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-api-access-nhzfq\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.661410 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661131 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-metrics-tls\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:38.661410 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661187 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f81e0175-b8d3-4348-bc49-d9e00531643f-ovn-node-metrics-cert\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.661410 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661245 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.661410 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661292 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysconfig\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.661410 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661350 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-var-lib-kubelet\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.661410 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661398 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wfgp\" (UniqueName: \"kubernetes.io/projected/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-kube-api-access-6wfgp\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:38.661705 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661427 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.661705 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661513 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.661705 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661535 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-run\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.661705 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661551 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-metrics-client-ca\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.661705 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661564 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-ovn\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.661705 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661652 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.661705 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661693 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661726 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661753 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-web-config\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661779 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjk47\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-kube-api-access-zjk47\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661806 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661831 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661861 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-hosts-file\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661887 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6hm\" (UniqueName: \"kubernetes.io/projected/ae8ba755-ec98-496c-a59a-4f4625fea774-kube-api-access-5m6hm\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661932 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-volume\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661934 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd701aaf-13ea-4699-9c01-5167947a614a-metrics-certs\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661963 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c6469de-dc6b-405f-b14f-cf765fd9d06f-konnectivity-ca\") pod \"konnectivity-agent-z28kx\" (UID: \"4c6469de-dc6b-405f-b14f-cf765fd9d06f\") " pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:52:38.662009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.661995 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-host\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662028 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-textfile\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662055 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6048b8aa-9e93-4214-8b5a-ed6663b632b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662094 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7j2\" (UniqueName: \"kubernetes.io/projected/f81e0175-b8d3-4348-bc49-d9e00531643f-kube-api-access-hk7j2\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662113 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-textfile\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662124 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662154 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c6469de-dc6b-405f-b14f-cf765fd9d06f-konnectivity-ca\") pod \"konnectivity-agent-z28kx\" (UID: \"4c6469de-dc6b-405f-b14f-cf765fd9d06f\") " pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662158 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662198 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-sys\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662311 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-sys\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662343 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662364 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662380 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662395 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-device-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662410 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-etc-tuned\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.662437 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662434 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-sys-fs\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662476 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-socket-dir-parent\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662541 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-bin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662545 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662571 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41e23004-aede-47df-a301-b880f169a803-etc-tuned\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662623 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxnx\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-kube-api-access-vbxnx\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662688 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config-out\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662741 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config-out\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662773 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36ec01c3-dcbe-4caf-baff-dcade7691e2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662814 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-config-volume\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662840 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-web-config\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.662887 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662864 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-systemd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662893 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-hostroot\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662922 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662955 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.662975 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-socket-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663007 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71685c84-43a3-4272-8d8b-d6c623f8f592-cert\") pod \"ingress-canary-stttr\" (UID: \"71685c84-43a3-4272-8d8b-d6c623f8f592\") " pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663016 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36ec01c3-dcbe-4caf-baff-dcade7691e2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663027 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663018 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663045 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtpvg\" (UniqueName: \"kubernetes.io/projected/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-kube-api-access-xtpvg\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663060 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbmv\" (UniqueName: \"kubernetes.io/projected/002bc527-a392-444a-85a5-2dbe005559f4-kube-api-access-6gbmv\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663081 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-registration-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663099 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663115 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663138 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-tmp-dir\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663177 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cnibin\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663206 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-etc-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.663213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663210 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-tmp-dir\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663231 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-node-log\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663256 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663302 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663319 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663334 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663354 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-script-lib\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663387 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvgg\" (UniqueName: \"kubernetes.io/projected/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-kube-api-access-pqvgg\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663410 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c6469de-dc6b-405f-b14f-cf765fd9d06f-agent-certs\") pod \"konnectivity-agent-z28kx\" (UID: \"4c6469de-dc6b-405f-b14f-cf765fd9d06f\") " pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663436 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663461 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663499 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcg4w\" (UniqueName: \"kubernetes.io/projected/b39d4538-dc20-4f6e-8170-4f5c0327cf69-kube-api-access-rcg4w\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663524 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-iptables-alerter-script\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663550 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-slash\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663577 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-var-lib-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663575 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-script-lib\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663622 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-etc-kubernetes\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663654 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plp8r\" (UniqueName: \"kubernetes.io/projected/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-kube-api-access-plp8r\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663684 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c6469de-dc6b-405f-b14f-cf765fd9d06f-agent-certs\") pod \"konnectivity-agent-z28kx\" (UID: \"4c6469de-dc6b-405f-b14f-cf765fd9d06f\") " pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663702 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-iptables-alerter-script\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663715 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-lib-modules\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663733 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-tmp-dir\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663748 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36ec01c3-dcbe-4caf-baff-dcade7691e2e-data-volume\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663764 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-kubelet-config\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663787 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-config\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663804 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-host-slash\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663825 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-tmp-dir\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:38.663819 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663835 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36ec01c3-dcbe-4caf-baff-dcade7691e2e-data-volume\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663855 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-out\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663880 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663897 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663944 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663948 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-out\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663956 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f81e0175-b8d3-4348-bc49-d9e00531643f-ovnkube-config\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663974 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b39d4538-dc20-4f6e-8170-4f5c0327cf69-serviceca\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.663995 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-systemd\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664015 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664050 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-bin\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664105 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-netns\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664139 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-daemon-config\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664181 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664304 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664338 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36ec01c3-dcbe-4caf-baff-dcade7691e2e-crio-socket\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664307 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-daemon-config\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664372 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664379 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-original-pull-secret\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664389 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-netd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664409 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664457 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9z9n\" (UniqueName: \"kubernetes.io/projected/cd701aaf-13ea-4699-9c01-5167947a614a-kube-api-access-w9z9n\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664496 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664518 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkn9t\" (UniqueName: \"kubernetes.io/projected/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-kube-api-access-jkn9t\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664537 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jvj\" (UniqueName: \"kubernetes.io/projected/6048b8aa-9e93-4214-8b5a-ed6663b632b5-kube-api-access-l9jvj\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664554 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664570 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-wtmp\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664594 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-systemd-units\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664630 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b39d4538-dc20-4f6e-8170-4f5c0327cf69-serviceca\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664642 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-system-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664710 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-multus-certs\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664734 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-conf\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664751 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-root\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664796 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.664850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.664833 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.697005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.696794 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:52:38.697005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.696863 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 19:52:38.697005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.696882 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 19:52:38.697005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.696900 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 19:52:38.697005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.696910 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 19:52:38.697005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.696974 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.697005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.696999 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qc22z\"" Nov 02 19:52:38.697484 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697125 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 19:52:38.697484 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697299 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 19:52:38.697484 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697340 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 19:52:38.697657 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697508 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 19:52:38.697657 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697472 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2uh2tiljffmk1\"" Nov 02 19:52:38.697657 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697552 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 19:52:38.697657 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697567 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:52:38.697657 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697647 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697658 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697672 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697684 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697710 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697725 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xjzch\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697726 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697766 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697777 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697782 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697816 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697826 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697833 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697860 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697883 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 19:52:38.697929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.697891 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 19:52:38.698715 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.698098 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 19:52:38.698715 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.698261 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ks6xc\"" Nov 02 19:52:38.698715 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.698486 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r6f2h\"" Nov 02 19:52:38.698715 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.698661 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvmjq\"" Nov 02 19:52:38.698715 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.698688 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:52:38.698955 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.698828 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 19:52:38.698955 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.698852 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:52:38.699084 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.698954 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 19:52:38.699128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.699111 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 19:52:38.700900 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.700833 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-tls\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.700900 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.700861 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.701129 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.701114 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.701317 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.701301 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.702405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702140 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.702405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702154 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.702405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702166 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.702405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702183 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-web-config\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.702405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702140 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:38.702405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702213 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-volume\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.702405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702223 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.702405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702247 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-metrics-tls\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:38.702405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702314 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-metrics-client-ca\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.702405 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702383 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.702869 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.702854 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.703037 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703020 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:38.703068 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703021 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.703146 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703134 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-web-config\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.703177 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703156 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-config-volume\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:38.703213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703179 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.703290 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703263 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71685c84-43a3-4272-8d8b-d6c623f8f592-cert\") pod \"ingress-canary-stttr\" (UID: \"71685c84-43a3-4272-8d8b-d6c623f8f592\") " pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:38.703449 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703432 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.703486 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703456 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703806 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703809 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.703855 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.704040 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.704219 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.704393 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.704619 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.704798 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.704955 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.705079 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.707298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.705227 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.744223 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.744193 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 19:52:38.745174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.744355 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 19:52:38.745174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.744521 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 19:52:38.745174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.744785 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzfq\" (UniqueName: \"kubernetes.io/projected/36ec01c3-dcbe-4caf-baff-dcade7691e2e-kube-api-access-nhzfq\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.745174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.744818 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sng9v\" (UniqueName: \"kubernetes.io/projected/41e23004-aede-47df-a301-b880f169a803-kube-api-access-sng9v\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.745174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.744989 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6hm\" (UniqueName: \"kubernetes.io/projected/ae8ba755-ec98-496c-a59a-4f4625fea774-kube-api-access-5m6hm\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.745174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.745079 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2f7\" (UniqueName: \"kubernetes.io/projected/71685c84-43a3-4272-8d8b-d6c623f8f592-kube-api-access-9s2f7\") pod \"ingress-canary-stttr\" (UID: \"71685c84-43a3-4272-8d8b-d6c623f8f592\") " pod="openshift-ingress-canary/ingress-canary-stttr" Nov 02 19:52:38.745174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.745081 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.745174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.745104 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wfgp\" (UniqueName: \"kubernetes.io/projected/9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1-kube-api-access-6wfgp\") pod \"dns-default-87h2b\" (UID: \"9dfeeb7e-97f0-4d57-b450-bd17b4dfc8b1\") " pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:38.745174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.745152 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9z9n\" (UniqueName: \"kubernetes.io/projected/cd701aaf-13ea-4699-9c01-5167947a614a-kube-api-access-w9z9n\") pod \"network-metrics-daemon-7nt8t\" (UID: \"cd701aaf-13ea-4699-9c01-5167947a614a\") " pod="openshift-multus/network-metrics-daemon-7nt8t" Nov 02 19:52:38.745637 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.745297 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jvj\" (UniqueName: \"kubernetes.io/projected/6048b8aa-9e93-4214-8b5a-ed6663b632b5-kube-api-access-l9jvj\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.745637 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.745400 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psrcq\" (UniqueName: \"kubernetes.io/projected/040d6f5e-129c-4665-ad54-957415093fa3-kube-api-access-psrcq\") pod \"network-check-target-zg4fd\" (UID: \"040d6f5e-129c-4665-ad54-957415093fa3\") " pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:38.745637 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.745619 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plp8r\" (UniqueName: \"kubernetes.io/projected/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-kube-api-access-plp8r\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.745789 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.745646 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcg4w\" (UniqueName: \"kubernetes.io/projected/b39d4538-dc20-4f6e-8170-4f5c0327cf69-kube-api-access-rcg4w\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:52:38.745789 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.745700 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbmv\" (UniqueName: \"kubernetes.io/projected/002bc527-a392-444a-85a5-2dbe005559f4-kube-api-access-6gbmv\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.745949 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.745851 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxnx\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-kube-api-access-vbxnx\") pod \"alertmanager-main-0\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:52:38.746041 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.746022 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7j2\" (UniqueName: \"kubernetes.io/projected/f81e0175-b8d3-4348-bc49-d9e00531643f-kube-api-access-hk7j2\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.746085 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.746055 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjk47\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-kube-api-access-zjk47\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.746115 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.746101 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkn9t\" (UniqueName: \"kubernetes.io/projected/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-kube-api-access-jkn9t\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:52:38.746174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.746158 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtpvg\" (UniqueName: \"kubernetes.io/projected/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-kube-api-access-xtpvg\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:52:38.746240 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.746225 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvgg\" (UniqueName: \"kubernetes.io/projected/8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98-kube-api-access-pqvgg\") pod \"prometheus-operator-9cd89b74c-dbx7g\" (UID: \"8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dbx7g" Nov 02 19:52:38.751561 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.751539 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.757611 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.757592 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:38.765257 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765230 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39d4538-dc20-4f6e-8170-4f5c0327cf69-host\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:52:38.765362 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765260 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-kubernetes\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.765362 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765297 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-kubelet\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.765362 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765317 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.765362 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765341 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-system-cni-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.765362 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765350 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b39d4538-dc20-4f6e-8170-4f5c0327cf69-host\") pod \"node-ca-rmpzb\" (UID: \"b39d4538-dc20-4f6e-8170-4f5c0327cf69\") " pod="openshift-image-registry/node-ca-rmpzb" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765364 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765365 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-kubelet\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765373 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-kubernetes\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765409 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765413 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-dbus\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765433 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765450 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-system-cni-dir\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765475 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-k8s-cni-cncf-io\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765494 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-dbus\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765525 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-kubelet\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765543 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-k8s-cni-cncf-io\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.765599 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765588 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-kubelet\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765607 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-conf-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765640 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-netns\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765658 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-log-socket\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765660 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-conf-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765673 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cnibin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765713 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-log-socket\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765722 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-netns\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765726 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysconfig\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765759 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-cnibin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765763 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-var-lib-kubelet\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765766 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysconfig\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765799 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-run\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765804 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-var-lib-kubelet\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765824 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-ovn\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765850 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765877 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-hosts-file\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765890 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-ovn\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765902 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-host\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765854 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-run\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765900 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765928 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-sys\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765948 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1e35ec0a-ad15-48bc-ac59-e08c8dec75a0-hosts-file\") pod \"node-resolver-nc9rw\" (UID: \"1e35ec0a-ad15-48bc-ac59-e08c8dec75a0\") " pod="openshift-dns/node-resolver-nc9rw" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.765993 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-host\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766022 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-sys\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766022 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-sys\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766051 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-sys\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766113 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-device-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766129 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-sys-fs\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.766124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766145 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-socket-dir-parent\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766163 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-bin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766179 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-systemd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766197 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-hostroot\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766232 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-sys-fs\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766239 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-hostroot\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766237 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-socket-dir-parent\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766236 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-device-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766259 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-bin\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766300 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-socket-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766284 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-systemd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766319 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-registration-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766336 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cnibin\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766344 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-socket-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766353 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-etc-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766366 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-node-log\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766381 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766385 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-registration-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766396 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766420 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-slash\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766423 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-node-log\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766432 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-etc-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766442 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-var-lib-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766445 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6048b8aa-9e93-4214-8b5a-ed6663b632b5-cnibin\") pod \"multus-additional-cni-plugins-bn5zl\" (UID: \"6048b8aa-9e93-4214-8b5a-ed6663b632b5\") " pod="openshift-multus/multus-additional-cni-plugins-bn5zl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766464 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-multus-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766475 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-var-lib-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766475 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-etc-kubernetes\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766489 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-slash\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766506 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-lib-modules\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766507 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-etc-kubernetes\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766524 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766537 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-kubelet-config\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766568 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-host-slash\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766587 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-systemd\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766599 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-lib-modules\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766605 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-bin\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766612 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76aa7437-fa4a-42a6-90d5-9043d9b8a71d-kubelet-config\") pod \"global-pull-secret-syncer-wkg2t\" (UID: \"76aa7437-fa4a-42a6-90d5-9043d9b8a71d\") " pod="kube-system/global-pull-secret-syncer-wkg2t" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766628 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e71e3e7-20f2-4dbd-bd77-1debbbdacf68-host-slash\") pod \"iptables-alerter-pvp2h\" (UID: \"5e71e3e7-20f2-4dbd-bd77-1debbbdacf68\") " pod="openshift-network-operator/iptables-alerter-pvp2h" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766645 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-bin\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766647 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-systemd\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766682 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-netns\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766706 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36ec01c3-dcbe-4caf-baff-dcade7691e2e-crio-socket\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766727 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-netns\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766729 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766756 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-run-openvswitch\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766765 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-netd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766789 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36ec01c3-dcbe-4caf-baff-dcade7691e2e-crio-socket\") pod \"insights-runtime-extractor-pst9l\" (UID: \"36ec01c3-dcbe-4caf-baff-dcade7691e2e\") " pod="openshift-insights/insights-runtime-extractor-pst9l" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766793 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-cni-netd\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766795 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-wtmp\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766826 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-node-exporter-wtmp\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766840 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-systemd-units\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766863 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-system-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766878 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-systemd-units\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766886 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-multus-certs\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766909 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-system-cni-dir\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766909 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-conf\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766930 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-root\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766948 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766962 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-run-multus-certs\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766973 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/002bc527-a392-444a-85a5-2dbe005559f4-root\") pod \"node-exporter-x4ttn\" (UID: \"002bc527-a392-444a-85a5-2dbe005559f4\") " pod="openshift-monitoring/node-exporter-x4ttn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766975 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-modprobe-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.766989 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f81e0175-b8d3-4348-bc49-d9e00531643f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7m6dl\" (UID: \"f81e0175-b8d3-4348-bc49-d9e00531643f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.767008 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.767018 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-sysctl-conf\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.767033 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-multus\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.767042 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41e23004-aede-47df-a301-b880f169a803-etc-modprobe-d\") pod \"tuned-2nprn\" (UID: \"41e23004-aede-47df-a301-b880f169a803\") " pod="openshift-cluster-node-tuning-operator/tuned-2nprn" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.767067 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ae8ba755-ec98-496c-a59a-4f4625fea774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8p7w\" (UID: \"ae8ba755-ec98-496c-a59a-4f4625fea774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8p7w" Nov 02 19:52:38.767510 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.767088 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2-host-var-lib-cni-multus\") pod \"multus-gspv7\" (UID: \"7fd8e8aa-3e6c-40de-bce8-3b6d3fc194d2\") " pod="openshift-multus/multus-gspv7" Nov 02 19:52:38.859070 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:38.859050 5399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:52:40.416362 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:40.416239 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:41.571757 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:41.571727 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerStarted","Data":"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857"} Nov 02 19:52:41.572128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:41.571791 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerStarted","Data":"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89"} Nov 02 19:52:41.572128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:41.571816 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerStarted","Data":"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d"} Nov 02 19:52:41.572128 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:41.571841 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerStarted","Data":"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1"} Nov 02 19:52:41.632390 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:41.632360 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-87h2b" Nov 02 19:52:41.859509 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:41.859429 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:41.874317 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:41.874287 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:52:47.772479 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:47.772398 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:52:48.163800 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:48.163771 5399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-111.us-west-2.compute.internal" event="NodeReady" Nov 02 19:52:48.235191 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:52:48.235157 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:52:48.436957 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:48.436885 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:52:48.437096 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:48.437083 5399 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:52:53.976336 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:52:53.976311 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7m6dl" Nov 02 19:53:00.128576 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.128541 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:53:00.129305 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.129238 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="alertmanager" containerID="cri-o://9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5" gracePeriod=120 Nov 02 19:53:00.130592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.129564 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy" containerID="cri-o://24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227" gracePeriod=120 Nov 02 19:53:00.130592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.129667 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy-web" containerID="cri-o://6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a" gracePeriod=120 Nov 02 19:53:00.130592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.129748 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="prom-label-proxy" containerID="cri-o://507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e" gracePeriod=120 Nov 02 19:53:00.130592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.129802 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="config-reloader" containerID="cri-o://fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0" gracePeriod=120 Nov 02 19:53:00.130592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.129849 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy-metric" containerID="cri-o://b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db" gracePeriod=120 Nov 02 19:53:00.637112 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.637078 5399 generic.go:356] "Generic (PLEG): container finished" podID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerID="507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e" exitCode=0 Nov 02 19:53:00.637112 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.637106 5399 generic.go:356] "Generic (PLEG): container finished" podID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerID="b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db" exitCode=0 Nov 02 19:53:00.637112 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.637113 5399 generic.go:356] "Generic (PLEG): container finished" podID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerID="24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227" exitCode=0 Nov 02 19:53:00.637112 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.637118 5399 generic.go:356] "Generic (PLEG): container finished" podID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerID="fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0" exitCode=0 Nov 02 19:53:00.637400 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.637128 5399 generic.go:356] "Generic (PLEG): container finished" podID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerID="9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5" exitCode=0 Nov 02 19:53:00.637400 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.637148 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerDied","Data":"507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e"} Nov 02 19:53:00.637400 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.637218 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerDied","Data":"b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db"} Nov 02 19:53:00.637400 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.637243 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerDied","Data":"24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227"} Nov 02 19:53:00.637400 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.637267 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerDied","Data":"fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0"} Nov 02 19:53:00.637400 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:00.637308 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerDied","Data":"9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5"} Nov 02 19:53:01.380262 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.380233 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.433174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433109 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-cluster-tls-config\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433141 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-trusted-ca-bundle\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433364 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433182 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-tls-assets\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-tls-assets\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433364 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433211 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-volume\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-volume\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433364 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433248 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-web-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-web-config\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433364 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433314 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-main-tls\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433579 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433375 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-out\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-out\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433579 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433402 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-main-db\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433579 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433433 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-kube-api-access-vbxnx\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-kube-api-access-vbxnx\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433579 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433469 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433579 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433508 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-web\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433579 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433542 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-metrics-client-ca\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433579 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433572 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\" (UID: \"94d53725-3f49-4c81-b68a-1a60a69a2fbc\") " Nov 02 19:53:01.433917 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433589 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-trusted-ca-bundle" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:01.433917 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.433783 5399 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.434930 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.434897 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-metrics-client-ca" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:01.436236 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.436207 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-main-db" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:01.436909 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.436874 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-out" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:01.437002 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.436908 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-metric" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:01.437067 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.437041 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-tls-assets" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:01.437158 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.437132 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-main-tls" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:01.437269 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.437251 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-volume" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:01.437345 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.437331 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-kube-api-access-vbxnx" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "kube-api-access-vbxnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:01.438025 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.438005 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-web" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:01.438581 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.438553 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:01.440837 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.440663 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-cluster-tls-config" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:01.446086 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.446063 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-web-config" pod "94d53725-3f49-4c81-b68a-1a60a69a2fbc" (UID: "94d53725-3f49-4c81-b68a-1a60a69a2fbc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:01.534576 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534544 5399 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-out\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534576 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534574 5399 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/94d53725-3f49-4c81-b68a-1a60a69a2fbc-alertmanager-main-db\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534588 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbxnx\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-kube-api-access-vbxnx\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534601 5399 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534611 5399 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534619 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94d53725-3f49-4c81-b68a-1a60a69a2fbc-metrics-client-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534628 5399 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534637 5399 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-cluster-tls-config\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534646 5399 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94d53725-3f49-4c81-b68a-1a60a69a2fbc-tls-assets\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534661 5399 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534673 5399 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-web-config\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.534756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.534681 5399 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/94d53725-3f49-4c81-b68a-1a60a69a2fbc-secret-alertmanager-main-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:01.644780 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.644747 5399 generic.go:356] "Generic (PLEG): container finished" podID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerID="6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a" exitCode=0 Nov 02 19:53:01.644943 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.644841 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.644943 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.644840 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerDied","Data":"6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a"} Nov 02 19:53:01.645026 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.644981 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"94d53725-3f49-4c81-b68a-1a60a69a2fbc","Type":"ContainerDied","Data":"26e67e5f5480c8e90868081c6f4fc876dd75f50e375542abc6d78ced2ffa2828"} Nov 02 19:53:01.645026 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.645015 5399 scope.go:122] "RemoveContainer" containerID="507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e" Nov 02 19:53:01.653426 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.653406 5399 scope.go:122] "RemoveContainer" containerID="b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db" Nov 02 19:53:01.661202 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.661179 5399 scope.go:122] "RemoveContainer" containerID="24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227" Nov 02 19:53:01.667984 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.667964 5399 scope.go:122] "RemoveContainer" containerID="6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a" Nov 02 19:53:01.676911 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.676890 5399 scope.go:122] "RemoveContainer" containerID="fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0" Nov 02 19:53:01.684992 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.684968 5399 scope.go:122] "RemoveContainer" containerID="9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5" Nov 02 19:53:01.691910 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.691890 5399 scope.go:122] "RemoveContainer" containerID="3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c" Nov 02 19:53:01.698850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.698831 5399 scope.go:122] "RemoveContainer" containerID="507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e" Nov 02 19:53:01.699113 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:01.699088 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e\": container with ID starting with 507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e not found: ID does not exist" containerID="507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e" Nov 02 19:53:01.699163 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.699123 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e"} err="failed to get container status \"507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e\": rpc error: code = NotFound desc = could not find container \"507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e\": container with ID starting with 507d35e77ed94c21c4c2beec9991b56e29abb02aa855f833051b14cfad72e16e not found: ID does not exist" Nov 02 19:53:01.699163 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.699154 5399 scope.go:122] "RemoveContainer" containerID="b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db" Nov 02 19:53:01.699423 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:01.699405 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db\": container with ID starting with b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db not found: ID does not exist" containerID="b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db" Nov 02 19:53:01.699477 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.699425 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db"} err="failed to get container status \"b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db\": rpc error: code = NotFound desc = could not find container \"b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db\": container with ID starting with b4d4833284ecdea4ef3e7864019a5cf1a544dab4dc8f7cff12a996aa7e7731db not found: ID does not exist" Nov 02 19:53:01.699477 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.699438 5399 scope.go:122] "RemoveContainer" containerID="24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227" Nov 02 19:53:01.699672 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:01.699657 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227\": container with ID starting with 24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227 not found: ID does not exist" containerID="24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227" Nov 02 19:53:01.699717 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.699675 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227"} err="failed to get container status \"24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227\": rpc error: code = NotFound desc = could not find container \"24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227\": container with ID starting with 24a51084b3069fec30e56069bcefff6e1a493c94d9f8af26dad8291268e18227 not found: ID does not exist" Nov 02 19:53:01.699717 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.699686 5399 scope.go:122] "RemoveContainer" containerID="6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a" Nov 02 19:53:01.699904 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:01.699889 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a\": container with ID starting with 6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a not found: ID does not exist" containerID="6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a" Nov 02 19:53:01.699952 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.699910 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a"} err="failed to get container status \"6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a\": rpc error: code = NotFound desc = could not find container \"6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a\": container with ID starting with 6ce9dde890213eeef1bb85ec4334b82f73f4405e175cd569074505c231bc6f1a not found: ID does not exist" Nov 02 19:53:01.699952 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.699922 5399 scope.go:122] "RemoveContainer" containerID="fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0" Nov 02 19:53:01.700136 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:01.700121 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0\": container with ID starting with fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0 not found: ID does not exist" containerID="fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0" Nov 02 19:53:01.700178 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.700141 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0"} err="failed to get container status \"fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0\": rpc error: code = NotFound desc = could not find container \"fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0\": container with ID starting with fa1b53d10ef8f818988aeef9155a8b369a98d6d7f4b702d5448e707bef59b1e0 not found: ID does not exist" Nov 02 19:53:01.700178 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.700152 5399 scope.go:122] "RemoveContainer" containerID="9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5" Nov 02 19:53:01.700397 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:01.700378 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5\": container with ID starting with 9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5 not found: ID does not exist" containerID="9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5" Nov 02 19:53:01.700475 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.700406 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5"} err="failed to get container status \"9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5\": rpc error: code = NotFound desc = could not find container \"9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5\": container with ID starting with 9e151404af1e2d30ce68785811ff92c01d074b1a1a945efeb1d35213850808e5 not found: ID does not exist" Nov 02 19:53:01.700475 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.700429 5399 scope.go:122] "RemoveContainer" containerID="3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c" Nov 02 19:53:01.700655 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:01.700635 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c\": container with ID starting with 3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c not found: ID does not exist" containerID="3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c" Nov 02 19:53:01.700699 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.700661 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c"} err="failed to get container status \"3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c\": rpc error: code = NotFound desc = could not find container \"3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c\": container with ID starting with 3b36e12a48bb184966fedcbfcd0180a54cbaf47f04ce5c821dd9dea143472f7c not found: ID does not exist" Nov 02 19:53:01.799244 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.799214 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:53:01.802859 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.802835 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:53:01.845352 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845159 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:53:01.845548 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845494 5399 cpu_manager.go:414] "containerMap: removing container" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="alertmanager" Nov 02 19:53:01.845548 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845510 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="alertmanager" Nov 02 19:53:01.845548 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845530 5399 cpu_manager.go:414] "containerMap: removing container" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy-web" Nov 02 19:53:01.845548 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845538 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy-web" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845553 5399 cpu_manager.go:414] "containerMap: removing container" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="config-reloader" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845562 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="config-reloader" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845576 5399 cpu_manager.go:414] "containerMap: removing container" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="prom-label-proxy" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845583 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="prom-label-proxy" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845597 5399 cpu_manager.go:414] "containerMap: removing container" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845603 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845611 5399 cpu_manager.go:414] "containerMap: removing container" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="init-config-reloader" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845617 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="init-config-reloader" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845635 5399 cpu_manager.go:414] "containerMap: removing container" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy-metric" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845642 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy-metric" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845701 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845711 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="init-config-reloader" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845721 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy-metric" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845732 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="alertmanager" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845741 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="kube-rbac-proxy-web" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845751 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="config-reloader" Nov 02 19:53:01.845772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.845759 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" containerName="prom-label-proxy" Nov 02 19:53:01.851944 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.851918 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.901031 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.900966 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 19:53:01.901320 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.901069 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 19:53:01.901320 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.901086 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 19:53:01.901320 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.901096 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 19:53:01.901320 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.901139 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 19:53:01.901320 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.901153 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 19:53:01.901320 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.901165 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 19:53:01.901320 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.901200 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ks6xc\"" Nov 02 19:53:01.901661 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.901306 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 19:53:01.937835 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.937740 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c86b88-12f0-4433-b67b-7c97e7670cdf-config-out\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.937835 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.937795 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938032 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.937904 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c86b88-12f0-4433-b67b-7c97e7670cdf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938032 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.937942 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938139 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.938043 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/93c86b88-12f0-4433-b67b-7c97e7670cdf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938139 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.938082 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-web-config\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938139 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.938109 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c86b88-12f0-4433-b67b-7c97e7670cdf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938290 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.938243 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93c86b88-12f0-4433-b67b-7c97e7670cdf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938361 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.938317 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-config-volume\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938410 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.938373 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938459 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.938412 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938509 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.938462 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx7nc\" (UniqueName: \"kubernetes.io/projected/93c86b88-12f0-4433-b67b-7c97e7670cdf-kube-api-access-hx7nc\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.938509 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.938491 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:01.947398 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.947372 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 19:53:01.947960 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:01.947941 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039545 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039630 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c86b88-12f0-4433-b67b-7c97e7670cdf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039669 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039709 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/93c86b88-12f0-4433-b67b-7c97e7670cdf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039745 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-web-config\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039778 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c86b88-12f0-4433-b67b-7c97e7670cdf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039830 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93c86b88-12f0-4433-b67b-7c97e7670cdf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039855 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-config-volume\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039895 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039935 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.039973 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx7nc\" (UniqueName: \"kubernetes.io/projected/93c86b88-12f0-4433-b67b-7c97e7670cdf-kube-api-access-hx7nc\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.040015 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.040043 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c86b88-12f0-4433-b67b-7c97e7670cdf-config-out\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.042942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.040532 5399 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:53:02.043795 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.043492 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/93c86b88-12f0-4433-b67b-7c97e7670cdf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.044223 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.044098 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c86b88-12f0-4433-b67b-7c97e7670cdf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.044392 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.044363 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93c86b88-12f0-4433-b67b-7c97e7670cdf-config-out\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.045301 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.045258 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93c86b88-12f0-4433-b67b-7c97e7670cdf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.046689 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.046653 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.047077 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.047055 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-web-config\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.047224 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.047179 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.047444 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.047421 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-config-volume\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.047543 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.047463 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93c86b88-12f0-4433-b67b-7c97e7670cdf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.047543 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.047505 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.047672 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.047574 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.047755 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.047737 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/93c86b88-12f0-4433-b67b-7c97e7670cdf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.095942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.095904 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx7nc\" (UniqueName: \"kubernetes.io/projected/93c86b88-12f0-4433-b67b-7c97e7670cdf-kube-api-access-hx7nc\") pod \"alertmanager-main-0\" (UID: \"93c86b88-12f0-4433-b67b-7c97e7670cdf\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.179084 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.179047 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 19:53:02.641354 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.641324 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 19:53:02.643686 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:53:02.643646 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c86b88_12f0_4433_b67b_7c97e7670cdf.slice/crio-f8143925b6856fb4b045545a6d3d479a84f56cf82314db7dfaab13d6823afdd5 WatchSource:0}: Error finding container f8143925b6856fb4b045545a6d3d479a84f56cf82314db7dfaab13d6823afdd5: Status 404 returned error can't find the container with id f8143925b6856fb4b045545a6d3d479a84f56cf82314db7dfaab13d6823afdd5 Nov 02 19:53:02.649918 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:02.649887 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93c86b88-12f0-4433-b67b-7c97e7670cdf","Type":"ContainerStarted","Data":"f8143925b6856fb4b045545a6d3d479a84f56cf82314db7dfaab13d6823afdd5"} Nov 02 19:53:03.392309 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.392262 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c566bd4cd-nbpxg"] Nov 02 19:53:03.395555 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.395539 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.446794 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.446760 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 19:53:03.446794 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.446780 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 19:53:03.447018 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.446779 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 19:53:03.447018 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.446813 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 19:53:03.447018 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.446879 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xmz5z\"" Nov 02 19:53:03.447018 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.447002 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 19:53:03.450192 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.450172 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.450253 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.450204 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3371f31-30e5-4c4c-9a08-e036e0b221b6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.450253 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.450225 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3371f31-30e5-4c4c-9a08-e036e0b221b6-serving-certs-ca-bundle\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.450344 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.450270 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-secret-telemeter-client\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.450381 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.450348 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-telemeter-client-tls\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.450428 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.450392 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3371f31-30e5-4c4c-9a08-e036e0b221b6-metrics-client-ca\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.450484 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.450465 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvx8z\" (UniqueName: \"kubernetes.io/projected/a3371f31-30e5-4c4c-9a08-e036e0b221b6-kube-api-access-rvx8z\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.450534 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.450511 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-federate-client-tls\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.504800 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.504767 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d53725-3f49-4c81-b68a-1a60a69a2fbc" path="/var/lib/kubelet/pods/94d53725-3f49-4c81-b68a-1a60a69a2fbc/volumes" Nov 02 19:53:03.540197 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.540163 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 19:53:03.540914 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.540892 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c566bd4cd-nbpxg"] Nov 02 19:53:03.551775 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.551753 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-secret-telemeter-client\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.551871 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.551782 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-telemeter-client-tls\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.551871 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.551815 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3371f31-30e5-4c4c-9a08-e036e0b221b6-metrics-client-ca\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.551871 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.551840 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvx8z\" (UniqueName: \"kubernetes.io/projected/a3371f31-30e5-4c4c-9a08-e036e0b221b6-kube-api-access-rvx8z\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.552029 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.551872 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-federate-client-tls\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.552029 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.551934 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.552029 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.551959 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3371f31-30e5-4c4c-9a08-e036e0b221b6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.552029 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.551986 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3371f31-30e5-4c4c-9a08-e036e0b221b6-serving-certs-ca-bundle\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.552714 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.552687 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3371f31-30e5-4c4c-9a08-e036e0b221b6-serving-certs-ca-bundle\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.552844 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.552736 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3371f31-30e5-4c4c-9a08-e036e0b221b6-metrics-client-ca\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.553211 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.553187 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3371f31-30e5-4c4c-9a08-e036e0b221b6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.554529 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.554503 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.554621 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.554527 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-federate-client-tls\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.554621 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.554580 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-secret-telemeter-client\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.554741 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.554688 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a3371f31-30e5-4c4c-9a08-e036e0b221b6-telemeter-client-tls\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.604855 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.604815 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvx8z\" (UniqueName: \"kubernetes.io/projected/a3371f31-30e5-4c4c-9a08-e036e0b221b6-kube-api-access-rvx8z\") pod \"telemeter-client-c566bd4cd-nbpxg\" (UID: \"a3371f31-30e5-4c4c-9a08-e036e0b221b6\") " pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:03.655661 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.655581 5399 generic.go:356] "Generic (PLEG): container finished" podID="93c86b88-12f0-4433-b67b-7c97e7670cdf" containerID="c250882b0899e6d6d5d2a836b345de640a2137aab08e841ad0ad3289c7ebdcd9" exitCode=0 Nov 02 19:53:03.655661 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.655642 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93c86b88-12f0-4433-b67b-7c97e7670cdf","Type":"ContainerDied","Data":"c250882b0899e6d6d5d2a836b345de640a2137aab08e841ad0ad3289c7ebdcd9"} Nov 02 19:53:03.710801 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:03.710776 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" Nov 02 19:53:04.122832 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:04.122801 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c566bd4cd-nbpxg"] Nov 02 19:53:04.124788 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:53:04.124759 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3371f31_30e5_4c4c_9a08_e036e0b221b6.slice/crio-f7fe764f3038b726ab1be064f77d8ca432b654185aed98833d2ea0d3c2ee9a86 WatchSource:0}: Error finding container f7fe764f3038b726ab1be064f77d8ca432b654185aed98833d2ea0d3c2ee9a86: Status 404 returned error can't find the container with id f7fe764f3038b726ab1be064f77d8ca432b654185aed98833d2ea0d3c2ee9a86 Nov 02 19:53:04.662432 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:04.662392 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93c86b88-12f0-4433-b67b-7c97e7670cdf","Type":"ContainerStarted","Data":"9097ea56712aa6e9c6a75b590b0aae88174db85a144aded4429f74f4dc2f09ee"} Nov 02 19:53:04.662865 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:04.662466 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93c86b88-12f0-4433-b67b-7c97e7670cdf","Type":"ContainerStarted","Data":"dde59682b5556ec92f0bdc5c2baceeaee0351a8309dcd1eef108122d8c03fe38"} Nov 02 19:53:04.662865 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:04.662510 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93c86b88-12f0-4433-b67b-7c97e7670cdf","Type":"ContainerStarted","Data":"b9e9c0e6d7d59d469488d51ff3c8edd3b809a4e85e6cf041f58b95126ccc6523"} Nov 02 19:53:04.662865 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:04.662541 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93c86b88-12f0-4433-b67b-7c97e7670cdf","Type":"ContainerStarted","Data":"d152e4ed5aab96cbcce60bd913181ecebb4305d30737cb7260e064d67a506df1"} Nov 02 19:53:04.662865 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:04.662565 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93c86b88-12f0-4433-b67b-7c97e7670cdf","Type":"ContainerStarted","Data":"4a18653cf5a1abf66c283d37ffb50b5f9a627a956bf12bb8b228729734f8078b"} Nov 02 19:53:04.662865 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:04.662601 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93c86b88-12f0-4433-b67b-7c97e7670cdf","Type":"ContainerStarted","Data":"ea35c73c840ec088e35d59be2b8a19f7659a3101911dd4bd389ba373c77d9c6a"} Nov 02 19:53:04.663597 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:04.663577 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" event={"ID":"a3371f31-30e5-4c4c-9a08-e036e0b221b6","Type":"ContainerStarted","Data":"f7fe764f3038b726ab1be064f77d8ca432b654185aed98833d2ea0d3c2ee9a86"} Nov 02 19:53:04.777960 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:04.777229 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.767047781 podStartE2EDuration="3.777211006s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.163225ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:04.776242685 +0000 UTC m=+27.386894776" watchObservedRunningTime="2026-11-02 19:53:04.777211006 +0000 UTC m=+27.387863098" Nov 02 19:53:05.406049 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.406003 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 19:53:05.406436 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.406404 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="prometheus" containerID="cri-o://2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080" gracePeriod=600 Nov 02 19:53:05.406752 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.406724 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy-thanos" containerID="cri-o://dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857" gracePeriod=600 Nov 02 19:53:05.406981 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.406829 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy" containerID="cri-o://5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89" gracePeriod=600 Nov 02 19:53:05.406981 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.406905 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy-web" containerID="cri-o://69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d" gracePeriod=600 Nov 02 19:53:05.406981 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.406977 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="thanos-sidecar" containerID="cri-o://41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1" gracePeriod=600 Nov 02 19:53:05.407150 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.407048 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="config-reloader" containerID="cri-o://a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879" gracePeriod=600 Nov 02 19:53:05.647214 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.647181 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z28kx" Nov 02 19:53:05.670397 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.670310 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:05.674788 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.674754 5399 generic.go:356] "Generic (PLEG): container finished" podID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerID="dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857" exitCode=0 Nov 02 19:53:05.674788 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.674783 5399 generic.go:356] "Generic (PLEG): container finished" podID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerID="5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89" exitCode=0 Nov 02 19:53:05.674951 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.674793 5399 generic.go:356] "Generic (PLEG): container finished" podID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerID="69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d" exitCode=0 Nov 02 19:53:05.674951 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.674805 5399 generic.go:356] "Generic (PLEG): container finished" podID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerID="41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1" exitCode=0 Nov 02 19:53:05.674951 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.674813 5399 generic.go:356] "Generic (PLEG): container finished" podID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerID="a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879" exitCode=0 Nov 02 19:53:05.674951 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.674823 5399 generic.go:356] "Generic (PLEG): container finished" podID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerID="2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080" exitCode=0 Nov 02 19:53:05.674951 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.674896 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerDied","Data":"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857"} Nov 02 19:53:05.675159 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.674973 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerDied","Data":"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89"} Nov 02 19:53:05.675159 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.675043 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerDied","Data":"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d"} Nov 02 19:53:05.675159 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.675052 5399 scope.go:122] "RemoveContainer" containerID="dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857" Nov 02 19:53:05.675159 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.675082 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerDied","Data":"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1"} Nov 02 19:53:05.675159 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.675127 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerDied","Data":"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879"} Nov 02 19:53:05.675384 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.675168 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerDied","Data":"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080"} Nov 02 19:53:05.675384 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.675320 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"75f8744c-1c79-4c6a-b2fb-c7aeac335710","Type":"ContainerDied","Data":"b3e47f32cb33b982fac5a898072a4791f9a1b54c4b50207a5c7587844802885c"} Nov 02 19:53:05.683710 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.683689 5399 scope.go:122] "RemoveContainer" containerID="5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89" Nov 02 19:53:05.691931 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.691915 5399 scope.go:122] "RemoveContainer" containerID="69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d" Nov 02 19:53:05.699533 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.699513 5399 scope.go:122] "RemoveContainer" containerID="41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1" Nov 02 19:53:05.707718 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.707696 5399 scope.go:122] "RemoveContainer" containerID="a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879" Nov 02 19:53:05.717578 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.717553 5399 scope.go:122] "RemoveContainer" containerID="2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080" Nov 02 19:53:05.725574 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.725551 5399 scope.go:122] "RemoveContainer" containerID="edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded" Nov 02 19:53:05.734287 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.734261 5399 scope.go:122] "RemoveContainer" containerID="dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857" Nov 02 19:53:05.734641 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:05.734591 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": container with ID starting with dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857 not found: ID does not exist" containerID="dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857" Nov 02 19:53:05.734732 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.734653 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857"} err="failed to get container status \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": rpc error: code = NotFound desc = could not find container \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": container with ID starting with dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857 not found: ID does not exist" Nov 02 19:53:05.734732 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.734680 5399 scope.go:122] "RemoveContainer" containerID="5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89" Nov 02 19:53:05.734976 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:05.734957 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": container with ID starting with 5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89 not found: ID does not exist" containerID="5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89" Nov 02 19:53:05.735035 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.734983 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89"} err="failed to get container status \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": rpc error: code = NotFound desc = could not find container \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": container with ID starting with 5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89 not found: ID does not exist" Nov 02 19:53:05.735035 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.735001 5399 scope.go:122] "RemoveContainer" containerID="69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d" Nov 02 19:53:05.735323 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:05.735300 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": container with ID starting with 69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d not found: ID does not exist" containerID="69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d" Nov 02 19:53:05.735403 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.735323 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d"} err="failed to get container status \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": rpc error: code = NotFound desc = could not find container \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": container with ID starting with 69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d not found: ID does not exist" Nov 02 19:53:05.735403 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.735345 5399 scope.go:122] "RemoveContainer" containerID="41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1" Nov 02 19:53:05.735575 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:05.735552 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": container with ID starting with 41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1 not found: ID does not exist" containerID="41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1" Nov 02 19:53:05.735627 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.735581 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1"} err="failed to get container status \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": rpc error: code = NotFound desc = could not find container \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": container with ID starting with 41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1 not found: ID does not exist" Nov 02 19:53:05.735627 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.735602 5399 scope.go:122] "RemoveContainer" containerID="a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879" Nov 02 19:53:05.735916 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:05.735899 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": container with ID starting with a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879 not found: ID does not exist" containerID="a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879" Nov 02 19:53:05.735994 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.735920 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879"} err="failed to get container status \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": rpc error: code = NotFound desc = could not find container \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": container with ID starting with a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879 not found: ID does not exist" Nov 02 19:53:05.735994 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.735933 5399 scope.go:122] "RemoveContainer" containerID="2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080" Nov 02 19:53:05.736187 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:05.736166 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": container with ID starting with 2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080 not found: ID does not exist" containerID="2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080" Nov 02 19:53:05.736228 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.736190 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080"} err="failed to get container status \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": rpc error: code = NotFound desc = could not find container \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": container with ID starting with 2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080 not found: ID does not exist" Nov 02 19:53:05.736228 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.736208 5399 scope.go:122] "RemoveContainer" containerID="edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded" Nov 02 19:53:05.736498 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:05.736477 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": container with ID starting with edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded not found: ID does not exist" containerID="edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded" Nov 02 19:53:05.736562 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.736502 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded"} err="failed to get container status \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": rpc error: code = NotFound desc = could not find container \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": container with ID starting with edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded not found: ID does not exist" Nov 02 19:53:05.736562 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.736515 5399 scope.go:122] "RemoveContainer" containerID="dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857" Nov 02 19:53:05.736787 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.736761 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857"} err="failed to get container status \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": rpc error: code = NotFound desc = could not find container \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": container with ID starting with dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857 not found: ID does not exist" Nov 02 19:53:05.736832 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.736790 5399 scope.go:122] "RemoveContainer" containerID="5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89" Nov 02 19:53:05.737143 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.737119 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89"} err="failed to get container status \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": rpc error: code = NotFound desc = could not find container \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": container with ID starting with 5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89 not found: ID does not exist" Nov 02 19:53:05.737229 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.737143 5399 scope.go:122] "RemoveContainer" containerID="69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d" Nov 02 19:53:05.737457 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.737429 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d"} err="failed to get container status \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": rpc error: code = NotFound desc = could not find container \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": container with ID starting with 69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d not found: ID does not exist" Nov 02 19:53:05.737528 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.737450 5399 scope.go:122] "RemoveContainer" containerID="41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1" Nov 02 19:53:05.737824 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.737793 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1"} err="failed to get container status \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": rpc error: code = NotFound desc = could not find container \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": container with ID starting with 41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1 not found: ID does not exist" Nov 02 19:53:05.737907 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.737826 5399 scope.go:122] "RemoveContainer" containerID="a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879" Nov 02 19:53:05.738103 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.738076 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879"} err="failed to get container status \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": rpc error: code = NotFound desc = could not find container \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": container with ID starting with a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879 not found: ID does not exist" Nov 02 19:53:05.738103 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.738102 5399 scope.go:122] "RemoveContainer" containerID="2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080" Nov 02 19:53:05.738383 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.738360 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080"} err="failed to get container status \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": rpc error: code = NotFound desc = could not find container \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": container with ID starting with 2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080 not found: ID does not exist" Nov 02 19:53:05.738383 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.738381 5399 scope.go:122] "RemoveContainer" containerID="edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded" Nov 02 19:53:05.738623 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.738601 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded"} err="failed to get container status \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": rpc error: code = NotFound desc = could not find container \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": container with ID starting with edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded not found: ID does not exist" Nov 02 19:53:05.738623 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.738624 5399 scope.go:122] "RemoveContainer" containerID="dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857" Nov 02 19:53:05.738907 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.738887 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857"} err="failed to get container status \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": rpc error: code = NotFound desc = could not find container \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": container with ID starting with dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857 not found: ID does not exist" Nov 02 19:53:05.738907 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.738907 5399 scope.go:122] "RemoveContainer" containerID="5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89" Nov 02 19:53:05.739170 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.739155 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89"} err="failed to get container status \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": rpc error: code = NotFound desc = could not find container \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": container with ID starting with 5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89 not found: ID does not exist" Nov 02 19:53:05.739228 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.739178 5399 scope.go:122] "RemoveContainer" containerID="69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d" Nov 02 19:53:05.739442 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.739422 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d"} err="failed to get container status \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": rpc error: code = NotFound desc = could not find container \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": container with ID starting with 69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d not found: ID does not exist" Nov 02 19:53:05.739530 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.739444 5399 scope.go:122] "RemoveContainer" containerID="41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1" Nov 02 19:53:05.739775 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.739746 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1"} err="failed to get container status \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": rpc error: code = NotFound desc = could not find container \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": container with ID starting with 41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1 not found: ID does not exist" Nov 02 19:53:05.739858 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.739776 5399 scope.go:122] "RemoveContainer" containerID="a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879" Nov 02 19:53:05.740040 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.740013 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879"} err="failed to get container status \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": rpc error: code = NotFound desc = could not find container \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": container with ID starting with a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879 not found: ID does not exist" Nov 02 19:53:05.740089 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.740042 5399 scope.go:122] "RemoveContainer" containerID="2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080" Nov 02 19:53:05.740341 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.740323 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080"} err="failed to get container status \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": rpc error: code = NotFound desc = could not find container \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": container with ID starting with 2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080 not found: ID does not exist" Nov 02 19:53:05.740395 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.740342 5399 scope.go:122] "RemoveContainer" containerID="edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded" Nov 02 19:53:05.740592 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.740574 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded"} err="failed to get container status \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": rpc error: code = NotFound desc = could not find container \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": container with ID starting with edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded not found: ID does not exist" Nov 02 19:53:05.740659 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.740592 5399 scope.go:122] "RemoveContainer" containerID="dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857" Nov 02 19:53:05.740827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.740802 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857"} err="failed to get container status \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": rpc error: code = NotFound desc = could not find container \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": container with ID starting with dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857 not found: ID does not exist" Nov 02 19:53:05.740885 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.740829 5399 scope.go:122] "RemoveContainer" containerID="5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89" Nov 02 19:53:05.741076 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.741059 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89"} err="failed to get container status \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": rpc error: code = NotFound desc = could not find container \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": container with ID starting with 5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89 not found: ID does not exist" Nov 02 19:53:05.741076 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.741075 5399 scope.go:122] "RemoveContainer" containerID="69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d" Nov 02 19:53:05.741354 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.741332 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d"} err="failed to get container status \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": rpc error: code = NotFound desc = could not find container \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": container with ID starting with 69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d not found: ID does not exist" Nov 02 19:53:05.741354 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.741354 5399 scope.go:122] "RemoveContainer" containerID="41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1" Nov 02 19:53:05.741597 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.741577 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1"} err="failed to get container status \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": rpc error: code = NotFound desc = could not find container \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": container with ID starting with 41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1 not found: ID does not exist" Nov 02 19:53:05.741675 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.741599 5399 scope.go:122] "RemoveContainer" containerID="a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879" Nov 02 19:53:05.741848 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.741830 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879"} err="failed to get container status \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": rpc error: code = NotFound desc = could not find container \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": container with ID starting with a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879 not found: ID does not exist" Nov 02 19:53:05.741921 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.741848 5399 scope.go:122] "RemoveContainer" containerID="2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080" Nov 02 19:53:05.742085 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.742068 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080"} err="failed to get container status \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": rpc error: code = NotFound desc = could not find container \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": container with ID starting with 2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080 not found: ID does not exist" Nov 02 19:53:05.742149 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.742085 5399 scope.go:122] "RemoveContainer" containerID="edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded" Nov 02 19:53:05.742382 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.742356 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded"} err="failed to get container status \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": rpc error: code = NotFound desc = could not find container \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": container with ID starting with edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded not found: ID does not exist" Nov 02 19:53:05.742382 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.742381 5399 scope.go:122] "RemoveContainer" containerID="dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857" Nov 02 19:53:05.742748 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.742720 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857"} err="failed to get container status \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": rpc error: code = NotFound desc = could not find container \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": container with ID starting with dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857 not found: ID does not exist" Nov 02 19:53:05.742748 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.742739 5399 scope.go:122] "RemoveContainer" containerID="5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89" Nov 02 19:53:05.742963 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.742940 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89"} err="failed to get container status \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": rpc error: code = NotFound desc = could not find container \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": container with ID starting with 5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89 not found: ID does not exist" Nov 02 19:53:05.743030 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.742974 5399 scope.go:122] "RemoveContainer" containerID="69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d" Nov 02 19:53:05.743253 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.743234 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d"} err="failed to get container status \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": rpc error: code = NotFound desc = could not find container \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": container with ID starting with 69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d not found: ID does not exist" Nov 02 19:53:05.743253 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.743252 5399 scope.go:122] "RemoveContainer" containerID="41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1" Nov 02 19:53:05.743559 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.743528 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1"} err="failed to get container status \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": rpc error: code = NotFound desc = could not find container \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": container with ID starting with 41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1 not found: ID does not exist" Nov 02 19:53:05.743559 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.743558 5399 scope.go:122] "RemoveContainer" containerID="a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879" Nov 02 19:53:05.743824 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.743796 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879"} err="failed to get container status \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": rpc error: code = NotFound desc = could not find container \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": container with ID starting with a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879 not found: ID does not exist" Nov 02 19:53:05.743824 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.743823 5399 scope.go:122] "RemoveContainer" containerID="2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080" Nov 02 19:53:05.744067 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.744050 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080"} err="failed to get container status \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": rpc error: code = NotFound desc = could not find container \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": container with ID starting with 2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080 not found: ID does not exist" Nov 02 19:53:05.744133 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.744068 5399 scope.go:122] "RemoveContainer" containerID="edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded" Nov 02 19:53:05.744303 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.744267 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded"} err="failed to get container status \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": rpc error: code = NotFound desc = could not find container \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": container with ID starting with edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded not found: ID does not exist" Nov 02 19:53:05.744303 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.744300 5399 scope.go:122] "RemoveContainer" containerID="dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857" Nov 02 19:53:05.744523 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.744501 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857"} err="failed to get container status \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": rpc error: code = NotFound desc = could not find container \"dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857\": container with ID starting with dade6f67029968a498976f95fcc716990389be3b395a2395d9b2a010b69b9857 not found: ID does not exist" Nov 02 19:53:05.744582 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.744524 5399 scope.go:122] "RemoveContainer" containerID="5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89" Nov 02 19:53:05.744791 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.744760 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89"} err="failed to get container status \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": rpc error: code = NotFound desc = could not find container \"5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89\": container with ID starting with 5f967cdbe594a29cf2e7f8b4e781b4a90998aede71d715c15befe9b3be790b89 not found: ID does not exist" Nov 02 19:53:05.744864 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.744791 5399 scope.go:122] "RemoveContainer" containerID="69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d" Nov 02 19:53:05.745025 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.745008 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d"} err="failed to get container status \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": rpc error: code = NotFound desc = could not find container \"69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d\": container with ID starting with 69c65c5fa86ce4e5dc0c0610ae848c1ba77f1506bdeb1eaabca9b7ba7b42d81d not found: ID does not exist" Nov 02 19:53:05.745066 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.745025 5399 scope.go:122] "RemoveContainer" containerID="41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1" Nov 02 19:53:05.745264 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.745237 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1"} err="failed to get container status \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": rpc error: code = NotFound desc = could not find container \"41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1\": container with ID starting with 41214f1068642ad59910203abeee4aa1297569b95e8666aa48ef05ff4f8c71a1 not found: ID does not exist" Nov 02 19:53:05.745264 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.745256 5399 scope.go:122] "RemoveContainer" containerID="a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879" Nov 02 19:53:05.745505 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.745487 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879"} err="failed to get container status \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": rpc error: code = NotFound desc = could not find container \"a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879\": container with ID starting with a21d65c405386b8ba2af39b9f138e965410dba9af588c5fc66159f5a3dbc7879 not found: ID does not exist" Nov 02 19:53:05.745565 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.745505 5399 scope.go:122] "RemoveContainer" containerID="2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080" Nov 02 19:53:05.745790 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.745772 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080"} err="failed to get container status \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": rpc error: code = NotFound desc = could not find container \"2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080\": container with ID starting with 2670a28b589bb6e7ea29ebc466c421dbce4378e94207556cafe4e734f09de080 not found: ID does not exist" Nov 02 19:53:05.745790 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.745790 5399 scope.go:122] "RemoveContainer" containerID="edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded" Nov 02 19:53:05.746064 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.746027 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded"} err="failed to get container status \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": rpc error: code = NotFound desc = could not find container \"edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded\": container with ID starting with edadd0aa072055ad4ce6de412a5be0590293fec94084bfde25589fb46070fded not found: ID does not exist" Nov 02 19:53:05.771183 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771157 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-kube-rbac-proxy\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771354 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771192 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-web-config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-web-config\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771354 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771220 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771354 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771261 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-trusted-ca-bundle\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771354 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771324 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-0\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771527 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771355 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-2\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771527 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771412 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config-out\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771527 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771436 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-kube-api-access-zjk47\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-kube-api-access-zjk47\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771527 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771493 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-tls-assets\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-tls-assets\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771701 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771528 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-1\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771701 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771553 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-metrics-client-ca\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771701 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771588 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-metrics-client-certs\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771701 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771609 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-thanos-prometheus-http-client-file\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771701 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771668 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-grpc-tls\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771943 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771700 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-db\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771943 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771747 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-kubelet-serving-ca-bundle\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771943 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771777 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771943 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771821 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771943 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771850 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-tls\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.771943 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.771877 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-serving-certs-ca-bundle\") pod \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\" (UID: \"75f8744c-1c79-4c6a-b2fb-c7aeac335710\") " Nov 02 19:53:05.773587 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.772766 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-0" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:05.773587 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.772806 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-kubelet-serving-ca-bundle" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:05.773769 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.773707 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-db" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:05.774994 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.773994 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-trusted-ca-bundle" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:05.774994 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.774143 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-2" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:05.774994 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.774529 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-1" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:05.774994 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.774683 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-serving-certs-ca-bundle" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:05.774994 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.774790 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-metrics-client-ca" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:53:05.775827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.775755 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-thanos-sidecar-tls" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:05.775963 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.775906 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-kube-rbac-proxy" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:05.776726 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.776705 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-kube-rbac-proxy-web" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:05.776953 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.776922 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config-out" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:05.777270 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.777242 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-grpc-tls" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:05.777270 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.777337 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-tls" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:05.777615 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.777452 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-thanos-prometheus-http-client-file" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:05.777615 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.777598 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:05.777940 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.777914 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-metrics-client-certs" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:05.778370 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.778344 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-kube-api-access-zjk47" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "kube-api-access-zjk47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:05.778767 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.778747 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-tls-assets" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:05.787682 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.787656 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-web-config" pod "75f8744c-1c79-4c6a-b2fb-c7aeac335710" (UID: "75f8744c-1c79-4c6a-b2fb-c7aeac335710"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:53:05.873021 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.872982 5399 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-kube-rbac-proxy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873021 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873021 5399 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-web-config\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873036 5399 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873046 5399 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873056 5399 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873065 5399 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873074 5399 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config-out\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873084 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjk47\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-kube-api-access-zjk47\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873093 5399 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f8744c-1c79-4c6a-b2fb-c7aeac335710-tls-assets\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873101 5399 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873110 5399 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-metrics-client-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873119 5399 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-metrics-client-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873127 5399 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873143 5399 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-grpc-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873152 5399 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/75f8744c-1c79-4c6a-b2fb-c7aeac335710-prometheus-k8s-db\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873166 5399 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873175 5399 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-config\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873186 5399 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873195 5399 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/75f8744c-1c79-4c6a-b2fb-c7aeac335710-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:05.873237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:05.873205 5399 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f8744c-1c79-4c6a-b2fb-c7aeac335710-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:06.683289 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.683244 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" event={"ID":"a3371f31-30e5-4c4c-9a08-e036e0b221b6","Type":"ContainerStarted","Data":"07d0028e9a1bb4a30fcea32013e83dab8685c869db4a6e6a643332e0bec291b9"} Nov 02 19:53:06.683655 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.683326 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" event={"ID":"a3371f31-30e5-4c4c-9a08-e036e0b221b6","Type":"ContainerStarted","Data":"12503b359cf4452d8077595044e5d121f43eb5a0d6890e60f2e383b12a68d9e5"} Nov 02 19:53:06.684229 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.684208 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:06.885378 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.885343 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 19:53:06.890105 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.890086 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 19:53:06.932110 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932077 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 19:53:06.932365 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932350 5399 cpu_manager.go:414] "containerMap: removing container" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy-thanos" Nov 02 19:53:06.932442 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932366 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy-thanos" Nov 02 19:53:06.932442 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932385 5399 cpu_manager.go:414] "containerMap: removing container" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="thanos-sidecar" Nov 02 19:53:06.932442 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932393 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="thanos-sidecar" Nov 02 19:53:06.932442 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932406 5399 cpu_manager.go:414] "containerMap: removing container" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy" Nov 02 19:53:06.932442 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932414 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy" Nov 02 19:53:06.932442 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932432 5399 cpu_manager.go:414] "containerMap: removing container" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy-web" Nov 02 19:53:06.932442 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932437 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy-web" Nov 02 19:53:06.932442 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932444 5399 cpu_manager.go:414] "containerMap: removing container" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="prometheus" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932450 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="prometheus" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932470 5399 cpu_manager.go:414] "containerMap: removing container" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="config-reloader" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932474 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="config-reloader" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932479 5399 cpu_manager.go:414] "containerMap: removing container" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="init-config-reloader" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932484 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="init-config-reloader" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932530 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="config-reloader" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932538 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy-thanos" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932545 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="thanos-sidecar" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932550 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932556 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="kube-rbac-proxy-web" Nov 02 19:53:06.932667 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.932562 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" containerName="prometheus" Nov 02 19:53:06.950189 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.950123 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.000012 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:06.999971 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 19:53:07.000181 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000041 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 19:53:07.000181 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000059 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 19:53:07.000181 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000099 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 19:53:07.000181 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000114 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2uh2tiljffmk1\"" Nov 02 19:53:07.000420 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000206 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 19:53:07.000420 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000219 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 19:53:07.000420 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000229 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 19:53:07.000420 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000245 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 19:53:07.000420 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000261 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 19:53:07.000420 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000324 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 19:53:07.000635 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000537 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r6f2h\"" Nov 02 19:53:07.000635 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000593 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 19:53:07.000827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000810 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 19:53:07.000888 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.000859 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 19:53:07.047213 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.047184 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 19:53:07.047372 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.047249 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 19:53:07.082029 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.081998 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082029 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082032 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-config\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082227 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082052 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082227 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082114 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082227 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082155 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082412 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082237 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5x4\" (UniqueName: \"kubernetes.io/projected/503bac07-a421-4771-966e-51cd80f926f4-kube-api-access-hh5x4\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082412 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082260 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-web-config\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082412 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082300 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082412 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082377 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082588 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082419 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082588 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082455 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/503bac07-a421-4771-966e-51cd80f926f4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082588 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082479 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082588 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082503 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082588 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082558 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082588 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082577 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082877 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082648 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082877 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082699 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/503bac07-a421-4771-966e-51cd80f926f4-config-out\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082877 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082726 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082877 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082784 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.082877 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.082821 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.150484 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.150441 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 19:53:07.183795 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.183756 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.183795 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.183795 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.184056 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.183817 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.184056 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.183839 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/503bac07-a421-4771-966e-51cd80f926f4-config-out\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.184056 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.183856 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.184056 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.183875 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.184056 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.183901 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184413 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184468 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-config\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184495 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184522 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184548 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184590 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5x4\" (UniqueName: \"kubernetes.io/projected/503bac07-a421-4771-966e-51cd80f926f4-kube-api-access-hh5x4\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184618 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-web-config\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184645 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184684 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184710 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184741 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/503bac07-a421-4771-966e-51cd80f926f4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184751 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184766 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184789 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184991 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.185517 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.185172 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.186397 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.186116 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.186449 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.186396 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.187242 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.187215 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.187782 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.184735 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.187874 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.187814 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.188202 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.188075 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.188378 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.188358 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.188543 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.188506 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/503bac07-a421-4771-966e-51cd80f926f4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.188785 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.188744 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-web-config\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.188961 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.188942 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/503bac07-a421-4771-966e-51cd80f926f4-config-out\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.188961 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.188955 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-config\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.189091 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.189041 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.189324 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.189306 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.189932 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.189910 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.190829 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.190809 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/503bac07-a421-4771-966e-51cd80f926f4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.191064 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.191047 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/503bac07-a421-4771-966e-51cd80f926f4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.238704 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.238631 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5x4\" (UniqueName: \"kubernetes.io/projected/503bac07-a421-4771-966e-51cd80f926f4-kube-api-access-hh5x4\") pod \"prometheus-k8s-0\" (UID: \"503bac07-a421-4771-966e-51cd80f926f4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.356866 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.356832 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:07.504785 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.504705 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f8744c-1c79-4c6a-b2fb-c7aeac335710" path="/var/lib/kubelet/pods/75f8744c-1c79-4c6a-b2fb-c7aeac335710/volumes" Nov 02 19:53:07.690078 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.690040 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" event={"ID":"a3371f31-30e5-4c4c-9a08-e036e0b221b6","Type":"ContainerStarted","Data":"cede547d2c04015b003e2aa152444ffaaccc4fe56d1083358be41f09530f3b42"} Nov 02 19:53:07.775707 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.775629 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zg4fd" Nov 02 19:53:07.800340 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.800121 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c566bd4cd-nbpxg" podStartSLOduration=2.4151595009999998 podStartE2EDuration="4.800104711s" totalImagesPullingTime="2.38494521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:07.79970124 +0000 UTC m=+30.410353332" watchObservedRunningTime="2026-11-02 19:53:07.800104711 +0000 UTC m=+30.410756796" Nov 02 19:53:07.844831 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:07.844795 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 19:53:07.850563 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:53:07.850531 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503bac07_a421_4771_966e_51cd80f926f4.slice/crio-8cd16622ec9fec8e54a451aa86eca5e62fed7f86f106eaf6232b2c214fed7f5b WatchSource:0}: Error finding container 8cd16622ec9fec8e54a451aa86eca5e62fed7f86f106eaf6232b2c214fed7f5b: Status 404 returned error can't find the container with id 8cd16622ec9fec8e54a451aa86eca5e62fed7f86f106eaf6232b2c214fed7f5b Nov 02 19:53:08.695447 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:08.695363 5399 generic.go:356] "Generic (PLEG): container finished" podID="503bac07-a421-4771-966e-51cd80f926f4" containerID="8cd05d874487e5c0da33c1fc8fc6b34ef554cd68b9da3df4afc19285e50fca4a" exitCode=0 Nov 02 19:53:08.695827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:08.695447 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"503bac07-a421-4771-966e-51cd80f926f4","Type":"ContainerDied","Data":"8cd05d874487e5c0da33c1fc8fc6b34ef554cd68b9da3df4afc19285e50fca4a"} Nov 02 19:53:08.695827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:08.695511 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"503bac07-a421-4771-966e-51cd80f926f4","Type":"ContainerStarted","Data":"8cd16622ec9fec8e54a451aa86eca5e62fed7f86f106eaf6232b2c214fed7f5b"} Nov 02 19:53:09.702064 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:09.702030 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"503bac07-a421-4771-966e-51cd80f926f4","Type":"ContainerStarted","Data":"98f541db712f38cf17383b40cc9244ef099c876a38a4261be2fa5bdb007cffee"} Nov 02 19:53:09.702496 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:09.702097 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"503bac07-a421-4771-966e-51cd80f926f4","Type":"ContainerStarted","Data":"5738b63fd9ad0b7a4c7e185537baa74d243ab440cabbfcfdb0a21e990aae64b9"} Nov 02 19:53:09.702496 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:09.702123 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"503bac07-a421-4771-966e-51cd80f926f4","Type":"ContainerStarted","Data":"ac8f1afae2cb46a29bddf5fb4a544ed2ecc85469715825fc846798f9af9fc664"} Nov 02 19:53:09.702496 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:09.702149 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"503bac07-a421-4771-966e-51cd80f926f4","Type":"ContainerStarted","Data":"9afa3c0422e1ea4e8e3e4981b53a7c678bbc95fea0e6d6e3815ee16f85650919"} Nov 02 19:53:09.702496 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:09.702174 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"503bac07-a421-4771-966e-51cd80f926f4","Type":"ContainerStarted","Data":"23c7a8d9011a28ea54e11d84c41e66acddb329cd2a153456ed270c71a78bcb1b"} Nov 02 19:53:09.702496 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:09.702195 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"503bac07-a421-4771-966e-51cd80f926f4","Type":"ContainerStarted","Data":"68fbdbb3ea34ff0cbf404c4f33b643159fb47e617f8f56e1cccfc4f562ca9d86"} Nov 02 19:53:12.357870 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:12.357830 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:18.234994 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:18.234956 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:53:33.813333 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:33.813271 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:33.828487 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:33.828458 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:34.082104 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:34.081986 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=28.057991768 podStartE2EDuration="28.081966341s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.974573ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:09.819729363 +0000 UTC m=+32.430381455" watchObservedRunningTime="2026-11-02 19:53:34.081966341 +0000 UTC m=+56.692618433" Nov 02 19:53:34.800256 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:53:34.800229 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 19:53:48.234388 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:53:48.234342 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:54:18.234479 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:54:18.234398 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:54:48.235434 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:54:48.235395 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:55:18.236015 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:55:18.235972 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:55:48.234472 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:55:48.234395 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:56:18.234747 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:56:18.234703 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:56:36.489465 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.489426 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26"] Nov 02 19:56:36.492952 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.492931 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.588091 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.588057 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-l7bdp\"" Nov 02 19:56:36.588091 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.588063 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 19:56:36.588315 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.588109 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 19:56:36.589007 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.588987 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26"] Nov 02 19:56:36.589662 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.589646 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r"] Nov 02 19:56:36.592826 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.592807 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.607154 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.607132 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74rj\" (UniqueName: \"kubernetes.io/projected/7a7f840f-d965-4e09-972e-2ca8847e96d9-kube-api-access-x74rj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.607256 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.607178 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.607512 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.607486 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.644100 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.644077 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-bnnqw\"" Nov 02 19:56:36.648400 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.648376 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r"] Nov 02 19:56:36.691221 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.691191 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q"] Nov 02 19:56:36.694559 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.694542 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:36.708043 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.708019 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.708162 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.708062 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74rj\" (UniqueName: \"kubernetes.io/projected/7a7f840f-d965-4e09-972e-2ca8847e96d9-kube-api-access-x74rj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.708162 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.708096 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.708162 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.708122 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.708162 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.708153 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6sgr\" (UniqueName: \"kubernetes.io/projected/3b511687-7be9-4696-b6a5-01894630787c-kube-api-access-q6sgr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.708420 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.708295 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.708481 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.708422 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.708536 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.708511 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.743890 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.743804 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-zfvwj\"" Nov 02 19:56:36.745129 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.745106 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q"] Nov 02 19:56:36.760847 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.760822 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74rj\" (UniqueName: \"kubernetes.io/projected/7a7f840f-d965-4e09-972e-2ca8847e96d9-kube-api-access-x74rj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.789153 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.789127 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p"] Nov 02 19:56:36.792380 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.792364 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:36.807594 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.807569 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:56:36.809078 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.809056 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.809130 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.809109 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:36.809164 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.809130 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:36.809164 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.809149 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.809225 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.809167 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6sgr\" (UniqueName: \"kubernetes.io/projected/3b511687-7be9-4696-b6a5-01894630787c-kube-api-access-q6sgr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.809406 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.809390 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.809468 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.809429 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwms7\" (UniqueName: \"kubernetes.io/projected/096ce13d-2fa8-4409-a55f-c85c0b98e4df-kube-api-access-pwms7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:36.809585 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.809566 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.840783 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.840754 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-mjj8f\"" Nov 02 19:56:36.845504 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.845438 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p"] Nov 02 19:56:36.862508 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.862452 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6sgr\" (UniqueName: \"kubernetes.io/projected/3b511687-7be9-4696-b6a5-01894630787c-kube-api-access-q6sgr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.908394 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.908366 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:56:36.910721 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.910700 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwms7\" (UniqueName: \"kubernetes.io/projected/096ce13d-2fa8-4409-a55f-c85c0b98e4df-kube-api-access-pwms7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:36.910785 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.910756 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:36.910821 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.910784 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ncst\" (UniqueName: \"kubernetes.io/projected/f6cfc31b-c326-4fac-b717-fe590a6eac7f-kube-api-access-9ncst\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:36.910821 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.910805 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:36.910885 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.910825 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:36.910885 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.910844 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:36.911199 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.911180 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:36.911233 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.911210 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:36.964788 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:36.964756 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwms7\" (UniqueName: \"kubernetes.io/projected/096ce13d-2fa8-4409-a55f-c85c0b98e4df-kube-api-access-pwms7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:37.010236 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.010156 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:56:37.011445 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.011422 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ncst\" (UniqueName: \"kubernetes.io/projected/f6cfc31b-c326-4fac-b717-fe590a6eac7f-kube-api-access-9ncst\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:37.011553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.011459 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:37.011553 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.011529 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:37.011898 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.011880 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:37.011935 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.011896 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:37.065719 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.065689 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ncst\" (UniqueName: \"kubernetes.io/projected/f6cfc31b-c326-4fac-b717-fe590a6eac7f-kube-api-access-9ncst\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:37.107987 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.107957 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:56:37.145454 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.145419 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd"] Nov 02 19:56:37.149036 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.149020 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.169043 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.169020 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26"] Nov 02 19:56:37.171732 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:56:37.171703 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7f840f_d965_4e09_972e_2ca8847e96d9.slice/crio-35e9bcbd7150b7cc7c9b97d9cdf02c1afaaad8e4c9d7b908b4bfbfd7f23e134c WatchSource:0}: Error finding container 35e9bcbd7150b7cc7c9b97d9cdf02c1afaaad8e4c9d7b908b4bfbfd7f23e134c: Status 404 returned error can't find the container with id 35e9bcbd7150b7cc7c9b97d9cdf02c1afaaad8e4c9d7b908b4bfbfd7f23e134c Nov 02 19:56:37.200838 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.200807 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-q7rjm\"" Nov 02 19:56:37.201812 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.201784 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd"] Nov 02 19:56:37.213140 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.213114 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.213268 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.213168 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q52wc\" (UniqueName: \"kubernetes.io/projected/abcc4deb-58a9-47f4-8b63-f755e2858a7a-kube-api-access-q52wc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.213268 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.213198 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.268223 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.268156 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r"] Nov 02 19:56:37.271575 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:56:37.271542 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b511687_7be9_4696_b6a5_01894630787c.slice/crio-5f5c19df1c3363593c340f63f135e48b4037c377eb6e09908de4793e31a60398 WatchSource:0}: Error finding container 5f5c19df1c3363593c340f63f135e48b4037c377eb6e09908de4793e31a60398: Status 404 returned error can't find the container with id 5f5c19df1c3363593c340f63f135e48b4037c377eb6e09908de4793e31a60398 Nov 02 19:56:37.314615 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.314590 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.314730 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.314634 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q52wc\" (UniqueName: \"kubernetes.io/projected/abcc4deb-58a9-47f4-8b63-f755e2858a7a-kube-api-access-q52wc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.314730 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.314664 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.315101 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.315078 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.315101 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.315093 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.366404 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.366374 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q"] Nov 02 19:56:37.369053 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.369030 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q52wc\" (UniqueName: \"kubernetes.io/projected/abcc4deb-58a9-47f4-8b63-f755e2858a7a-kube-api-access-q52wc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.369438 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:56:37.369415 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096ce13d_2fa8_4409_a55f_c85c0b98e4df.slice/crio-ad6a0086c16cba1a32f4ddde8a384e83238a1ac4c1f27209519d48a39d8fb056 WatchSource:0}: Error finding container ad6a0086c16cba1a32f4ddde8a384e83238a1ac4c1f27209519d48a39d8fb056: Status 404 returned error can't find the container with id ad6a0086c16cba1a32f4ddde8a384e83238a1ac4c1f27209519d48a39d8fb056 Nov 02 19:56:37.391770 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.391739 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" event={"ID":"096ce13d-2fa8-4409-a55f-c85c0b98e4df","Type":"ContainerStarted","Data":"ad6a0086c16cba1a32f4ddde8a384e83238a1ac4c1f27209519d48a39d8fb056"} Nov 02 19:56:37.392872 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.392848 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" event={"ID":"3b511687-7be9-4696-b6a5-01894630787c","Type":"ContainerStarted","Data":"5f5c19df1c3363593c340f63f135e48b4037c377eb6e09908de4793e31a60398"} Nov 02 19:56:37.393824 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.393805 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" event={"ID":"7a7f840f-d965-4e09-972e-2ca8847e96d9","Type":"ContainerStarted","Data":"35e9bcbd7150b7cc7c9b97d9cdf02c1afaaad8e4c9d7b908b4bfbfd7f23e134c"} Nov 02 19:56:37.465224 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.465192 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p"] Nov 02 19:56:37.467931 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:56:37.467903 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cfc31b_c326_4fac_b717_fe590a6eac7f.slice/crio-e315bbbb8e28894e5c88e7f693e002c8e3521e2af173b7e29d2cff63a2150a61 WatchSource:0}: Error finding container e315bbbb8e28894e5c88e7f693e002c8e3521e2af173b7e29d2cff63a2150a61: Status 404 returned error can't find the container with id e315bbbb8e28894e5c88e7f693e002c8e3521e2af173b7e29d2cff63a2150a61 Nov 02 19:56:37.516329 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.516299 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-q7rjm\"" Nov 02 19:56:37.519167 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.519122 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:56:37.880005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:37.879974 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd"] Nov 02 19:56:37.883048 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:56:37.883019 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcc4deb_58a9_47f4_8b63_f755e2858a7a.slice/crio-b6ac32584638cb6253f6ae4203c4dffd54e825f725f07f319f904a2e768e4ae4 WatchSource:0}: Error finding container b6ac32584638cb6253f6ae4203c4dffd54e825f725f07f319f904a2e768e4ae4: Status 404 returned error can't find the container with id b6ac32584638cb6253f6ae4203c4dffd54e825f725f07f319f904a2e768e4ae4 Nov 02 19:56:38.401847 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:38.401808 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" event={"ID":"f6cfc31b-c326-4fac-b717-fe590a6eac7f","Type":"ContainerStarted","Data":"e315bbbb8e28894e5c88e7f693e002c8e3521e2af173b7e29d2cff63a2150a61"} Nov 02 19:56:38.404474 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:38.404441 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" event={"ID":"abcc4deb-58a9-47f4-8b63-f755e2858a7a","Type":"ContainerStarted","Data":"b6ac32584638cb6253f6ae4203c4dffd54e825f725f07f319f904a2e768e4ae4"} Nov 02 19:56:39.842314 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:39.842270 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn"] Nov 02 19:56:39.846013 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:39.845986 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:39.897374 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:39.897341 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn"] Nov 02 19:56:39.937028 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:39.936990 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:39.937210 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:39.937048 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:39.937210 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:39.937179 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9nw\" (UniqueName: \"kubernetes.io/projected/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-kube-api-access-ng9nw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:39.942840 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:39.942741 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-kf5ws\"" Nov 02 19:56:40.039772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:40.038519 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9nw\" (UniqueName: \"kubernetes.io/projected/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-kube-api-access-ng9nw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:40.039772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:40.038625 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:40.039772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:40.038656 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:40.039772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:40.039111 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:40.039772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:40.039674 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:40.100496 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:40.100366 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9nw\" (UniqueName: \"kubernetes.io/projected/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-kube-api-access-ng9nw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:40.167766 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:40.167411 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:56:40.568446 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:40.568389 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn"] Nov 02 19:56:40.573904 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:56:40.573875 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86c816c_8ee5_4671_9964_cfc0c20cdc3a.slice/crio-b7efe3fdb6843001f4c862c92b00ffb5dfda7358ca0b4f05c01b30f88adfad5d WatchSource:0}: Error finding container b7efe3fdb6843001f4c862c92b00ffb5dfda7358ca0b4f05c01b30f88adfad5d: Status 404 returned error can't find the container with id b7efe3fdb6843001f4c862c92b00ffb5dfda7358ca0b4f05c01b30f88adfad5d Nov 02 19:56:41.434421 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:41.434325 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" event={"ID":"c86c816c-8ee5-4671-9964-cfc0c20cdc3a","Type":"ContainerStarted","Data":"b7efe3fdb6843001f4c862c92b00ffb5dfda7358ca0b4f05c01b30f88adfad5d"} Nov 02 19:56:48.270493 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:56:48.270379 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:56:50.494233 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:50.494148 5399 generic.go:356] "Generic (PLEG): container finished" podID="7a7f840f-d965-4e09-972e-2ca8847e96d9" containerID="f8176bc4c29418a1e41e15ffa10e2b250925d2ea57d6ac8c9b2c5b17e26dd418" exitCode=0 Nov 02 19:56:50.494733 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:50.494246 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" event={"ID":"7a7f840f-d965-4e09-972e-2ca8847e96d9","Type":"ContainerDied","Data":"f8176bc4c29418a1e41e15ffa10e2b250925d2ea57d6ac8c9b2c5b17e26dd418"} Nov 02 19:56:50.496074 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:50.495943 5399 generic.go:356] "Generic (PLEG): container finished" podID="abcc4deb-58a9-47f4-8b63-f755e2858a7a" containerID="6fad17b3b4d91491971e60c28a8a0d4ceae9bbb71d5b179a8ab547e32da66a2e" exitCode=0 Nov 02 19:56:50.496176 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:50.496106 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" event={"ID":"abcc4deb-58a9-47f4-8b63-f755e2858a7a","Type":"ContainerDied","Data":"6fad17b3b4d91491971e60c28a8a0d4ceae9bbb71d5b179a8ab547e32da66a2e"} Nov 02 19:56:50.497820 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:50.497795 5399 generic.go:356] "Generic (PLEG): container finished" podID="096ce13d-2fa8-4409-a55f-c85c0b98e4df" containerID="8d70df81b91788721434054ad272bb683fc106015de2578169d64efe6dc356b0" exitCode=0 Nov 02 19:56:50.497907 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:50.497862 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" event={"ID":"096ce13d-2fa8-4409-a55f-c85c0b98e4df","Type":"ContainerDied","Data":"8d70df81b91788721434054ad272bb683fc106015de2578169d64efe6dc356b0"} Nov 02 19:56:50.499743 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:50.499720 5399 generic.go:356] "Generic (PLEG): container finished" podID="3b511687-7be9-4696-b6a5-01894630787c" containerID="a1902ecf8fe3837e2cca4e722806b8816a02aad779d14733c9e56e62c508add0" exitCode=0 Nov 02 19:56:50.499812 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:50.499784 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" event={"ID":"3b511687-7be9-4696-b6a5-01894630787c","Type":"ContainerDied","Data":"a1902ecf8fe3837e2cca4e722806b8816a02aad779d14733c9e56e62c508add0"} Nov 02 19:56:51.510600 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:51.510560 5399 generic.go:356] "Generic (PLEG): container finished" podID="f6cfc31b-c326-4fac-b717-fe590a6eac7f" containerID="ea081f6312b9d81c30592545c726d04c8f57a4b6bda85a593652fa0a98078faa" exitCode=0 Nov 02 19:56:51.511047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:51.510625 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" event={"ID":"f6cfc31b-c326-4fac-b717-fe590a6eac7f","Type":"ContainerDied","Data":"ea081f6312b9d81c30592545c726d04c8f57a4b6bda85a593652fa0a98078faa"} Nov 02 19:56:52.515512 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:52.515478 5399 generic.go:356] "Generic (PLEG): container finished" podID="7a7f840f-d965-4e09-972e-2ca8847e96d9" containerID="a111d7810c9d0e41cdb62c5bc1f7567fcb9807715602bbe9188929324ecaa992" exitCode=0 Nov 02 19:56:52.515879 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:52.515551 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" event={"ID":"7a7f840f-d965-4e09-972e-2ca8847e96d9","Type":"ContainerDied","Data":"a111d7810c9d0e41cdb62c5bc1f7567fcb9807715602bbe9188929324ecaa992"} Nov 02 19:56:52.517649 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:52.517621 5399 generic.go:356] "Generic (PLEG): container finished" podID="abcc4deb-58a9-47f4-8b63-f755e2858a7a" containerID="cf5d1c00e2e188d5ef644d7756523da9de897d720a37020190a18945c6000338" exitCode=0 Nov 02 19:56:52.517741 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:52.517711 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" event={"ID":"abcc4deb-58a9-47f4-8b63-f755e2858a7a","Type":"ContainerDied","Data":"cf5d1c00e2e188d5ef644d7756523da9de897d720a37020190a18945c6000338"} Nov 02 19:56:52.519734 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:52.519711 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" event={"ID":"096ce13d-2fa8-4409-a55f-c85c0b98e4df","Type":"ContainerStarted","Data":"65f663c9397568f5e0b8d194e5caa7686489d1f3a47dc026c0e411ac01577152"} Nov 02 19:56:52.521348 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:52.521324 5399 generic.go:356] "Generic (PLEG): container finished" podID="c86c816c-8ee5-4671-9964-cfc0c20cdc3a" containerID="da76654953b6d537a1437a65ea84e073ab94aa47ddcd5c87b77e166792af0213" exitCode=0 Nov 02 19:56:52.521454 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:52.521347 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" event={"ID":"c86c816c-8ee5-4671-9964-cfc0c20cdc3a","Type":"ContainerDied","Data":"da76654953b6d537a1437a65ea84e073ab94aa47ddcd5c87b77e166792af0213"} Nov 02 19:56:52.523085 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:52.523064 5399 generic.go:356] "Generic (PLEG): container finished" podID="3b511687-7be9-4696-b6a5-01894630787c" containerID="8f0b2ea80cb9e962ff990aa515baa682b393292d9c97f0f2676ffbba66e5d930" exitCode=0 Nov 02 19:56:52.523195 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:52.523102 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" event={"ID":"3b511687-7be9-4696-b6a5-01894630787c","Type":"ContainerDied","Data":"8f0b2ea80cb9e962ff990aa515baa682b393292d9c97f0f2676ffbba66e5d930"} Nov 02 19:56:53.531200 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:53.530317 5399 generic.go:356] "Generic (PLEG): container finished" podID="096ce13d-2fa8-4409-a55f-c85c0b98e4df" containerID="65f663c9397568f5e0b8d194e5caa7686489d1f3a47dc026c0e411ac01577152" exitCode=0 Nov 02 19:56:53.531200 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:53.530387 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" event={"ID":"096ce13d-2fa8-4409-a55f-c85c0b98e4df","Type":"ContainerDied","Data":"65f663c9397568f5e0b8d194e5caa7686489d1f3a47dc026c0e411ac01577152"} Nov 02 19:56:54.538290 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:54.538244 5399 generic.go:356] "Generic (PLEG): container finished" podID="f6cfc31b-c326-4fac-b717-fe590a6eac7f" containerID="1096e2b099b58dd0674e7195ffd7c0a426258ad31654d8d1ce5763059318de8b" exitCode=0 Nov 02 19:56:54.538831 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:54.538334 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" event={"ID":"f6cfc31b-c326-4fac-b717-fe590a6eac7f","Type":"ContainerDied","Data":"1096e2b099b58dd0674e7195ffd7c0a426258ad31654d8d1ce5763059318de8b"} Nov 02 19:56:58.570692 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:58.569438 5399 generic.go:356] "Generic (PLEG): container finished" podID="c86c816c-8ee5-4671-9964-cfc0c20cdc3a" containerID="d20c537e5813ce1fbbff4c8c20c32c5214f4c2a51e3d9541caaf2bcca44829ed" exitCode=0 Nov 02 19:56:58.570692 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:56:58.569485 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" event={"ID":"c86c816c-8ee5-4671-9964-cfc0c20cdc3a","Type":"ContainerDied","Data":"d20c537e5813ce1fbbff4c8c20c32c5214f4c2a51e3d9541caaf2bcca44829ed"} Nov 02 19:57:05.615960 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:05.615860 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" event={"ID":"3b511687-7be9-4696-b6a5-01894630787c","Type":"ContainerStarted","Data":"da9ee461d7fe75e68517a33651f7eaf5e3e126d0a2d758663467b82d942960fc"} Nov 02 19:57:05.619197 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:05.619156 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" event={"ID":"7a7f840f-d965-4e09-972e-2ca8847e96d9","Type":"ContainerStarted","Data":"fcddff7d2311d2b047af461e9349c76b0a94bf675bb8d3fa2cd4cb35b9e7ffd2"} Nov 02 19:57:05.622618 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:05.622075 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" event={"ID":"abcc4deb-58a9-47f4-8b63-f755e2858a7a","Type":"ContainerStarted","Data":"2472e134cc66f1e9e934c448404c17170d627e798da09d7897cbaae7d92a41aa"} Nov 02 19:57:05.759312 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:05.754867 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" podStartSLOduration=3.073068098 podStartE2EDuration="29.754847496s" totalImagesPullingTime="26.681779398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:05.753614672 +0000 UTC m=+268.364266765" watchObservedRunningTime="2026-11-02 19:57:05.754847496 +0000 UTC m=+268.365499589" Nov 02 19:57:05.864463 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:05.864394 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" podStartSLOduration=3.008678165 podStartE2EDuration="29.86437576s" totalImagesPullingTime="26.855697595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:05.863415772 +0000 UTC m=+268.474067866" watchObservedRunningTime="2026-11-02 19:57:05.86437576 +0000 UTC m=+268.475027853" Nov 02 19:57:05.970634 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:05.970576 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" podStartSLOduration=2.469206646 podStartE2EDuration="28.970553763s" totalImagesPullingTime="26.501347117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:05.969515024 +0000 UTC m=+268.580167142" watchObservedRunningTime="2026-11-02 19:57:05.970553763 +0000 UTC m=+268.581205878" Nov 02 19:57:06.629291 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:06.629240 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" event={"ID":"c86c816c-8ee5-4671-9964-cfc0c20cdc3a","Type":"ContainerStarted","Data":"97021c6bb8b02e8783de846072f7eccae20e7d4ce6b6ee5526ff033976646518"} Nov 02 19:57:06.631691 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:06.631650 5399 generic.go:356] "Generic (PLEG): container finished" podID="3b511687-7be9-4696-b6a5-01894630787c" containerID="da9ee461d7fe75e68517a33651f7eaf5e3e126d0a2d758663467b82d942960fc" exitCode=0 Nov 02 19:57:06.631816 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:06.631723 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" event={"ID":"3b511687-7be9-4696-b6a5-01894630787c","Type":"ContainerDied","Data":"da9ee461d7fe75e68517a33651f7eaf5e3e126d0a2d758663467b82d942960fc"} Nov 02 19:57:06.634170 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:06.634141 5399 generic.go:356] "Generic (PLEG): container finished" podID="7a7f840f-d965-4e09-972e-2ca8847e96d9" containerID="fcddff7d2311d2b047af461e9349c76b0a94bf675bb8d3fa2cd4cb35b9e7ffd2" exitCode=0 Nov 02 19:57:06.634301 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:06.634224 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" event={"ID":"7a7f840f-d965-4e09-972e-2ca8847e96d9","Type":"ContainerDied","Data":"fcddff7d2311d2b047af461e9349c76b0a94bf675bb8d3fa2cd4cb35b9e7ffd2"} Nov 02 19:57:06.636628 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:06.636602 5399 generic.go:356] "Generic (PLEG): container finished" podID="abcc4deb-58a9-47f4-8b63-f755e2858a7a" containerID="2472e134cc66f1e9e934c448404c17170d627e798da09d7897cbaae7d92a41aa" exitCode=0 Nov 02 19:57:06.636725 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:06.636651 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" event={"ID":"abcc4deb-58a9-47f4-8b63-f755e2858a7a","Type":"ContainerDied","Data":"2472e134cc66f1e9e934c448404c17170d627e798da09d7897cbaae7d92a41aa"} Nov 02 19:57:06.737595 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:06.737544 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" podStartSLOduration=3.484324317 podStartE2EDuration="27.737525443s" totalImagesPullingTime="24.253201126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:56:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:06.737307923 +0000 UTC m=+269.347960016" watchObservedRunningTime="2026-11-02 19:57:06.737525443 +0000 UTC m=+269.348177537" Nov 02 19:57:07.642631 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.642600 5399 generic.go:356] "Generic (PLEG): container finished" podID="096ce13d-2fa8-4409-a55f-c85c0b98e4df" containerID="32726ef2767f138404aad4ff1e6683e7d29a71d2b3a2f2a82664d02e5a046021" exitCode=0 Nov 02 19:57:07.643068 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.642681 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" event={"ID":"096ce13d-2fa8-4409-a55f-c85c0b98e4df","Type":"ContainerDied","Data":"32726ef2767f138404aad4ff1e6683e7d29a71d2b3a2f2a82664d02e5a046021"} Nov 02 19:57:07.644769 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.644747 5399 generic.go:356] "Generic (PLEG): container finished" podID="f6cfc31b-c326-4fac-b717-fe590a6eac7f" containerID="97ca16b684ba855e3853f761f5e8a34f446b619f91ad29d335afd70ec44c1f95" exitCode=0 Nov 02 19:57:07.644883 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.644824 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" event={"ID":"f6cfc31b-c326-4fac-b717-fe590a6eac7f","Type":"ContainerDied","Data":"97ca16b684ba855e3853f761f5e8a34f446b619f91ad29d335afd70ec44c1f95"} Nov 02 19:57:07.646688 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.646667 5399 generic.go:356] "Generic (PLEG): container finished" podID="c86c816c-8ee5-4671-9964-cfc0c20cdc3a" containerID="97021c6bb8b02e8783de846072f7eccae20e7d4ce6b6ee5526ff033976646518" exitCode=0 Nov 02 19:57:07.646759 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.646741 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" event={"ID":"c86c816c-8ee5-4671-9964-cfc0c20cdc3a","Type":"ContainerDied","Data":"97021c6bb8b02e8783de846072f7eccae20e7d4ce6b6ee5526ff033976646518"} Nov 02 19:57:07.817290 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.817260 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:57:07.824148 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.824128 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:57:07.825067 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.825050 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:57:07.900485 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.900449 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-util\" (UniqueName: \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-util\") pod \"3b511687-7be9-4696-b6a5-01894630787c\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " Nov 02 19:57:07.900673 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.900503 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-util\" (UniqueName: \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-util\") pod \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " Nov 02 19:57:07.900673 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.900557 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b511687-7be9-4696-b6a5-01894630787c-kube-api-access-q6sgr\" (UniqueName: \"kubernetes.io/projected/3b511687-7be9-4696-b6a5-01894630787c-kube-api-access-q6sgr\") pod \"3b511687-7be9-4696-b6a5-01894630787c\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " Nov 02 19:57:07.900673 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.900594 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-util\" (UniqueName: \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-util\") pod \"7a7f840f-d965-4e09-972e-2ca8847e96d9\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " Nov 02 19:57:07.900673 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.900623 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7f840f-d965-4e09-972e-2ca8847e96d9-kube-api-access-x74rj\" (UniqueName: \"kubernetes.io/projected/7a7f840f-d965-4e09-972e-2ca8847e96d9-kube-api-access-x74rj\") pod \"7a7f840f-d965-4e09-972e-2ca8847e96d9\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " Nov 02 19:57:07.900673 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.900654 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-bundle\") pod \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " Nov 02 19:57:07.900921 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.900692 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-bundle\") pod \"7a7f840f-d965-4e09-972e-2ca8847e96d9\" (UID: \"7a7f840f-d965-4e09-972e-2ca8847e96d9\") " Nov 02 19:57:07.900921 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.900723 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-bundle\") pod \"3b511687-7be9-4696-b6a5-01894630787c\" (UID: \"3b511687-7be9-4696-b6a5-01894630787c\") " Nov 02 19:57:07.900921 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.900781 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abcc4deb-58a9-47f4-8b63-f755e2858a7a-kube-api-access-q52wc\" (UniqueName: \"kubernetes.io/projected/abcc4deb-58a9-47f4-8b63-f755e2858a7a-kube-api-access-q52wc\") pod \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\" (UID: \"abcc4deb-58a9-47f4-8b63-f755e2858a7a\") " Nov 02 19:57:07.901356 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.901317 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-bundle" pod "abcc4deb-58a9-47f4-8b63-f755e2858a7a" (UID: "abcc4deb-58a9-47f4-8b63-f755e2858a7a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:07.901937 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.901903 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-bundle" pod "7a7f840f-d965-4e09-972e-2ca8847e96d9" (UID: "7a7f840f-d965-4e09-972e-2ca8847e96d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:07.902167 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.902140 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-bundle" pod "3b511687-7be9-4696-b6a5-01894630787c" (UID: "3b511687-7be9-4696-b6a5-01894630787c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:07.903326 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.903305 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b511687-7be9-4696-b6a5-01894630787c-kube-api-access-q6sgr" pod "3b511687-7be9-4696-b6a5-01894630787c" (UID: "3b511687-7be9-4696-b6a5-01894630787c"). InnerVolumeSpecName "kube-api-access-q6sgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:07.903450 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.903426 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcc4deb-58a9-47f4-8b63-f755e2858a7a-kube-api-access-q52wc" pod "abcc4deb-58a9-47f4-8b63-f755e2858a7a" (UID: "abcc4deb-58a9-47f4-8b63-f755e2858a7a"). InnerVolumeSpecName "kube-api-access-q52wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:07.903655 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.903634 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7f840f-d965-4e09-972e-2ca8847e96d9-kube-api-access-x74rj" pod "7a7f840f-d965-4e09-972e-2ca8847e96d9" (UID: "7a7f840f-d965-4e09-972e-2ca8847e96d9"). InnerVolumeSpecName "kube-api-access-x74rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:07.905187 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.905165 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-util" pod "3b511687-7be9-4696-b6a5-01894630787c" (UID: "3b511687-7be9-4696-b6a5-01894630787c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:07.905918 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.905896 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-util" pod "7a7f840f-d965-4e09-972e-2ca8847e96d9" (UID: "7a7f840f-d965-4e09-972e-2ca8847e96d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:07.906435 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:07.906415 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-util" pod "abcc4deb-58a9-47f4-8b63-f755e2858a7a" (UID: "abcc4deb-58a9-47f4-8b63-f755e2858a7a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:08.001445 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.001410 5399 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-bundle\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:08.001445 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.001438 5399 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-bundle\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:08.001445 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.001448 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q52wc\" (UniqueName: \"kubernetes.io/projected/abcc4deb-58a9-47f4-8b63-f755e2858a7a-kube-api-access-q52wc\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:08.001677 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.001456 5399 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b511687-7be9-4696-b6a5-01894630787c-util\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:08.001677 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.001466 5399 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-util\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:08.001677 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.001475 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6sgr\" (UniqueName: \"kubernetes.io/projected/3b511687-7be9-4696-b6a5-01894630787c-kube-api-access-q6sgr\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:08.001677 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.001482 5399 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a7f840f-d965-4e09-972e-2ca8847e96d9-util\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:08.001677 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.001493 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x74rj\" (UniqueName: \"kubernetes.io/projected/7a7f840f-d965-4e09-972e-2ca8847e96d9-kube-api-access-x74rj\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:08.001677 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.001501 5399 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abcc4deb-58a9-47f4-8b63-f755e2858a7a-bundle\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:08.652539 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.652500 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" event={"ID":"7a7f840f-d965-4e09-972e-2ca8847e96d9","Type":"ContainerDied","Data":"35e9bcbd7150b7cc7c9b97d9cdf02c1afaaad8e4c9d7b908b4bfbfd7f23e134c"} Nov 02 19:57:08.653011 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.652569 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35e9bcbd7150b7cc7c9b97d9cdf02c1afaaad8e4c9d7b908b4bfbfd7f23e134c" Nov 02 19:57:08.653011 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.652518 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847at4x26" Nov 02 19:57:08.654379 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.654362 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" Nov 02 19:57:08.654511 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.654396 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mbgd" event={"ID":"abcc4deb-58a9-47f4-8b63-f755e2858a7a","Type":"ContainerDied","Data":"b6ac32584638cb6253f6ae4203c4dffd54e825f725f07f319f904a2e768e4ae4"} Nov 02 19:57:08.654511 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.654441 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6ac32584638cb6253f6ae4203c4dffd54e825f725f07f319f904a2e768e4ae4" Nov 02 19:57:08.656363 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.656341 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" Nov 02 19:57:08.656468 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.656388 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cq7s9r" event={"ID":"3b511687-7be9-4696-b6a5-01894630787c","Type":"ContainerDied","Data":"5f5c19df1c3363593c340f63f135e48b4037c377eb6e09908de4793e31a60398"} Nov 02 19:57:08.656468 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.656433 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f5c19df1c3363593c340f63f135e48b4037c377eb6e09908de4793e31a60398" Nov 02 19:57:08.825334 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.825308 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:57:08.827636 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.827615 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:57:08.831189 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.831170 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:57:08.907615 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.907540 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096ce13d-2fa8-4409-a55f-c85c0b98e4df-kube-api-access-pwms7\" (UniqueName: \"kubernetes.io/projected/096ce13d-2fa8-4409-a55f-c85c0b98e4df-kube-api-access-pwms7\") pod \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " Nov 02 19:57:08.907615 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.907580 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-kube-api-access-ng9nw\" (UniqueName: \"kubernetes.io/projected/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-kube-api-access-ng9nw\") pod \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " Nov 02 19:57:08.907615 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.907606 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6cfc31b-c326-4fac-b717-fe590a6eac7f-kube-api-access-9ncst\" (UniqueName: \"kubernetes.io/projected/f6cfc31b-c326-4fac-b717-fe590a6eac7f-kube-api-access-9ncst\") pod \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " Nov 02 19:57:08.907866 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.907826 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-bundle\") pod \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " Nov 02 19:57:08.907958 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.907911 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-bundle\" (UniqueName: \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-bundle\") pod \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " Nov 02 19:57:08.908034 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.907960 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-util\" (UniqueName: \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-util\") pod \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\" (UID: \"c86c816c-8ee5-4671-9964-cfc0c20cdc3a\") " Nov 02 19:57:08.908034 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.908000 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-util\" (UniqueName: \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-util\") pod \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " Nov 02 19:57:08.908137 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.908064 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-bundle\") pod \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\" (UID: \"f6cfc31b-c326-4fac-b717-fe590a6eac7f\") " Nov 02 19:57:08.908137 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.908099 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-util\" (UniqueName: \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-util\") pod \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\" (UID: \"096ce13d-2fa8-4409-a55f-c85c0b98e4df\") " Nov 02 19:57:08.909029 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.908215 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-bundle" pod "c86c816c-8ee5-4671-9964-cfc0c20cdc3a" (UID: "c86c816c-8ee5-4671-9964-cfc0c20cdc3a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:08.909029 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.908566 5399 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-bundle\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:08.909029 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.908996 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-bundle" pod "f6cfc31b-c326-4fac-b717-fe590a6eac7f" (UID: "f6cfc31b-c326-4fac-b717-fe590a6eac7f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:08.910267 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.910234 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cfc31b-c326-4fac-b717-fe590a6eac7f-kube-api-access-9ncst" pod "f6cfc31b-c326-4fac-b717-fe590a6eac7f" (UID: "f6cfc31b-c326-4fac-b717-fe590a6eac7f"). InnerVolumeSpecName "kube-api-access-9ncst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:08.910397 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.910290 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-kube-api-access-ng9nw" pod "c86c816c-8ee5-4671-9964-cfc0c20cdc3a" (UID: "c86c816c-8ee5-4671-9964-cfc0c20cdc3a"). InnerVolumeSpecName "kube-api-access-ng9nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:08.911004 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.910982 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-bundle" pod "096ce13d-2fa8-4409-a55f-c85c0b98e4df" (UID: "096ce13d-2fa8-4409-a55f-c85c0b98e4df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:08.911146 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.911128 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096ce13d-2fa8-4409-a55f-c85c0b98e4df-kube-api-access-pwms7" pod "096ce13d-2fa8-4409-a55f-c85c0b98e4df" (UID: "096ce13d-2fa8-4409-a55f-c85c0b98e4df"). InnerVolumeSpecName "kube-api-access-pwms7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:57:08.915935 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.915909 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-util" pod "f6cfc31b-c326-4fac-b717-fe590a6eac7f" (UID: "f6cfc31b-c326-4fac-b717-fe590a6eac7f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:08.916888 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.916863 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-util" pod "096ce13d-2fa8-4409-a55f-c85c0b98e4df" (UID: "096ce13d-2fa8-4409-a55f-c85c0b98e4df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:08.916960 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:08.916867 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-util" pod "c86c816c-8ee5-4671-9964-cfc0c20cdc3a" (UID: "c86c816c-8ee5-4671-9964-cfc0c20cdc3a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:57:09.008917 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.008884 5399 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-bundle\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:09.008917 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.008910 5399 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-util\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:09.008917 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.008921 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwms7\" (UniqueName: \"kubernetes.io/projected/096ce13d-2fa8-4409-a55f-c85c0b98e4df-kube-api-access-pwms7\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:09.008917 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.008931 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng9nw\" (UniqueName: \"kubernetes.io/projected/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-kube-api-access-ng9nw\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:09.009171 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.008941 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ncst\" (UniqueName: \"kubernetes.io/projected/f6cfc31b-c326-4fac-b717-fe590a6eac7f-kube-api-access-9ncst\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:09.009171 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.008950 5399 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/096ce13d-2fa8-4409-a55f-c85c0b98e4df-bundle\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:09.009171 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.008959 5399 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c86c816c-8ee5-4671-9964-cfc0c20cdc3a-util\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:09.009171 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.008966 5399 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cfc31b-c326-4fac-b717-fe590a6eac7f-util\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:57:09.662266 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.662230 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" event={"ID":"096ce13d-2fa8-4409-a55f-c85c0b98e4df","Type":"ContainerDied","Data":"ad6a0086c16cba1a32f4ddde8a384e83238a1ac4c1f27209519d48a39d8fb056"} Nov 02 19:57:09.662683 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.662297 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad6a0086c16cba1a32f4ddde8a384e83238a1ac4c1f27209519d48a39d8fb056" Nov 02 19:57:09.662683 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.662243 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c678d2q" Nov 02 19:57:09.664495 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.664307 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" event={"ID":"f6cfc31b-c326-4fac-b717-fe590a6eac7f","Type":"ContainerDied","Data":"e315bbbb8e28894e5c88e7f693e002c8e3521e2af173b7e29d2cff63a2150a61"} Nov 02 19:57:09.664495 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.664332 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcc6p" Nov 02 19:57:09.664495 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.664354 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e315bbbb8e28894e5c88e7f693e002c8e3521e2af173b7e29d2cff63a2150a61" Nov 02 19:57:09.666469 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.666446 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" event={"ID":"c86c816c-8ee5-4671-9964-cfc0c20cdc3a","Type":"ContainerDied","Data":"b7efe3fdb6843001f4c862c92b00ffb5dfda7358ca0b4f05c01b30f88adfad5d"} Nov 02 19:57:09.666578 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.666483 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7efe3fdb6843001f4c862c92b00ffb5dfda7358ca0b4f05c01b30f88adfad5d" Nov 02 19:57:09.666578 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:09.666502 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kbprn" Nov 02 19:57:11.801391 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801359 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp"] Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801628 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f6cfc31b-c326-4fac-b717-fe590a6eac7f" containerName="pull" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801638 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cfc31b-c326-4fac-b717-fe590a6eac7f" containerName="pull" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801650 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c86c816c-8ee5-4671-9964-cfc0c20cdc3a" containerName="pull" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801655 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86c816c-8ee5-4671-9964-cfc0c20cdc3a" containerName="pull" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801662 5399 cpu_manager.go:414] "containerMap: removing container" podUID="abcc4deb-58a9-47f4-8b63-f755e2858a7a" containerName="extract" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801667 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcc4deb-58a9-47f4-8b63-f755e2858a7a" containerName="extract" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801675 5399 cpu_manager.go:414] "containerMap: removing container" podUID="096ce13d-2fa8-4409-a55f-c85c0b98e4df" containerName="pull" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801679 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096ce13d-2fa8-4409-a55f-c85c0b98e4df" containerName="pull" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801689 5399 cpu_manager.go:414] "containerMap: removing container" podUID="096ce13d-2fa8-4409-a55f-c85c0b98e4df" containerName="util" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801693 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096ce13d-2fa8-4409-a55f-c85c0b98e4df" containerName="util" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801699 5399 cpu_manager.go:414] "containerMap: removing container" podUID="abcc4deb-58a9-47f4-8b63-f755e2858a7a" containerName="util" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801703 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcc4deb-58a9-47f4-8b63-f755e2858a7a" containerName="util" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801709 5399 cpu_manager.go:414] "containerMap: removing container" podUID="abcc4deb-58a9-47f4-8b63-f755e2858a7a" containerName="pull" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801713 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcc4deb-58a9-47f4-8b63-f755e2858a7a" containerName="pull" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801720 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3b511687-7be9-4696-b6a5-01894630787c" containerName="util" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801725 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b511687-7be9-4696-b6a5-01894630787c" containerName="util" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801729 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f6cfc31b-c326-4fac-b717-fe590a6eac7f" containerName="extract" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801733 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cfc31b-c326-4fac-b717-fe590a6eac7f" containerName="extract" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801746 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7a7f840f-d965-4e09-972e-2ca8847e96d9" containerName="extract" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801750 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7f840f-d965-4e09-972e-2ca8847e96d9" containerName="extract" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801759 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c86c816c-8ee5-4671-9964-cfc0c20cdc3a" containerName="extract" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801764 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86c816c-8ee5-4671-9964-cfc0c20cdc3a" containerName="extract" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801771 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7a7f840f-d965-4e09-972e-2ca8847e96d9" containerName="pull" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801775 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7f840f-d965-4e09-972e-2ca8847e96d9" containerName="pull" Nov 02 19:57:11.801773 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801784 5399 cpu_manager.go:414] "containerMap: removing container" podUID="096ce13d-2fa8-4409-a55f-c85c0b98e4df" containerName="extract" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801788 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096ce13d-2fa8-4409-a55f-c85c0b98e4df" containerName="extract" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801795 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c86c816c-8ee5-4671-9964-cfc0c20cdc3a" containerName="util" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801799 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86c816c-8ee5-4671-9964-cfc0c20cdc3a" containerName="util" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801804 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3b511687-7be9-4696-b6a5-01894630787c" containerName="extract" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801809 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b511687-7be9-4696-b6a5-01894630787c" containerName="extract" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801816 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f6cfc31b-c326-4fac-b717-fe590a6eac7f" containerName="util" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801820 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cfc31b-c326-4fac-b717-fe590a6eac7f" containerName="util" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801827 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3b511687-7be9-4696-b6a5-01894630787c" containerName="pull" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801831 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b511687-7be9-4696-b6a5-01894630787c" containerName="pull" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801837 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7a7f840f-d965-4e09-972e-2ca8847e96d9" containerName="util" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801841 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7f840f-d965-4e09-972e-2ca8847e96d9" containerName="util" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801878 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6cfc31b-c326-4fac-b717-fe590a6eac7f" containerName="extract" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801885 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7f840f-d965-4e09-972e-2ca8847e96d9" containerName="extract" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801891 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86c816c-8ee5-4671-9964-cfc0c20cdc3a" containerName="extract" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801898 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="096ce13d-2fa8-4409-a55f-c85c0b98e4df" containerName="extract" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801903 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b511687-7be9-4696-b6a5-01894630787c" containerName="extract" Nov 02 19:57:11.802531 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.801909 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="abcc4deb-58a9-47f4-8b63-f755e2858a7a" containerName="extract" Nov 02 19:57:11.806401 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.806374 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" Nov 02 19:57:11.902118 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.902082 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 19:57:11.902431 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.902410 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:57:11.902530 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.902451 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-m7jxb\"" Nov 02 19:57:11.903499 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.903479 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp"] Nov 02 19:57:11.933910 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.933882 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lm9h\" (UniqueName: \"kubernetes.io/projected/27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f-kube-api-access-8lm9h\") pod \"jobset-operator-674cd59dc5-bmpgp\" (UID: \"27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" Nov 02 19:57:11.934020 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:11.933967 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f-tmp\") pod \"jobset-operator-674cd59dc5-bmpgp\" (UID: \"27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" Nov 02 19:57:12.034864 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:12.034821 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f-tmp\") pod \"jobset-operator-674cd59dc5-bmpgp\" (UID: \"27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" Nov 02 19:57:12.035055 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:12.034871 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lm9h\" (UniqueName: \"kubernetes.io/projected/27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f-kube-api-access-8lm9h\") pod \"jobset-operator-674cd59dc5-bmpgp\" (UID: \"27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" Nov 02 19:57:12.035209 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:12.035187 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f-tmp\") pod \"jobset-operator-674cd59dc5-bmpgp\" (UID: \"27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" Nov 02 19:57:12.088219 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:12.088196 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lm9h\" (UniqueName: \"kubernetes.io/projected/27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f-kube-api-access-8lm9h\") pod \"jobset-operator-674cd59dc5-bmpgp\" (UID: \"27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" Nov 02 19:57:12.135322 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:12.135271 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" Nov 02 19:57:12.495767 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:12.495740 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp"] Nov 02 19:57:12.498716 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:57:12.498689 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27b4aeb3_bb1d_47a7_bcc7_9f7206791d1f.slice/crio-64fd6d815bff681a77ab5f2e6f0558baafb5c2bf57d2dd7f596dd5e569b3d644 WatchSource:0}: Error finding container 64fd6d815bff681a77ab5f2e6f0558baafb5c2bf57d2dd7f596dd5e569b3d644: Status 404 returned error can't find the container with id 64fd6d815bff681a77ab5f2e6f0558baafb5c2bf57d2dd7f596dd5e569b3d644 Nov 02 19:57:12.679325 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:12.679270 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" event={"ID":"27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f","Type":"ContainerStarted","Data":"64fd6d815bff681a77ab5f2e6f0558baafb5c2bf57d2dd7f596dd5e569b3d644"} Nov 02 19:57:15.693506 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:15.693469 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" event={"ID":"27b4aeb3-bb1d-47a7-bcc7-9f7206791d1f","Type":"ContainerStarted","Data":"3bfeb9b9d2166d205b9ad4e32d93d14152f873cd60a11425e3013fa9ab6a91b8"} Nov 02 19:57:15.802991 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:15.802891 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bmpgp" podStartSLOduration=1.77334828 podStartE2EDuration="4.802876206s" totalImagesPullingTime="3.029527926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:15.801899144 +0000 UTC m=+278.412551234" watchObservedRunningTime="2026-11-02 19:57:15.802876206 +0000 UTC m=+278.413528296" Nov 02 19:57:18.234873 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:57:18.234794 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:57:19.720075 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.720034 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6"] Nov 02 19:57:19.723244 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.723227 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" Nov 02 19:57:19.774801 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.774771 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6"] Nov 02 19:57:19.796364 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.796332 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jttpz\" (UniqueName: \"kubernetes.io/projected/b3fe0bf0-a9ee-4779-9343-c0f8f0012190-kube-api-access-jttpz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6\" (UID: \"b3fe0bf0-a9ee-4779-9343-c0f8f0012190\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" Nov 02 19:57:19.796506 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.796369 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3fe0bf0-a9ee-4779-9343-c0f8f0012190-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6\" (UID: \"b3fe0bf0-a9ee-4779-9343-c0f8f0012190\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" Nov 02 19:57:19.819998 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.819970 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 19:57:19.820160 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.820017 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-4glbf\"" Nov 02 19:57:19.820160 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.820034 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:57:19.897232 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.897199 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttpz\" (UniqueName: \"kubernetes.io/projected/b3fe0bf0-a9ee-4779-9343-c0f8f0012190-kube-api-access-jttpz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6\" (UID: \"b3fe0bf0-a9ee-4779-9343-c0f8f0012190\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" Nov 02 19:57:19.897232 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.897239 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3fe0bf0-a9ee-4779-9343-c0f8f0012190-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6\" (UID: \"b3fe0bf0-a9ee-4779-9343-c0f8f0012190\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" Nov 02 19:57:19.897595 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.897569 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3fe0bf0-a9ee-4779-9343-c0f8f0012190-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6\" (UID: \"b3fe0bf0-a9ee-4779-9343-c0f8f0012190\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" Nov 02 19:57:19.957998 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:19.957958 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jttpz\" (UniqueName: \"kubernetes.io/projected/b3fe0bf0-a9ee-4779-9343-c0f8f0012190-kube-api-access-jttpz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6\" (UID: \"b3fe0bf0-a9ee-4779-9343-c0f8f0012190\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" Nov 02 19:57:20.034894 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:20.034805 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" Nov 02 19:57:20.489310 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:20.489287 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6"] Nov 02 19:57:20.491542 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:57:20.491517 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fe0bf0_a9ee_4779_9343_c0f8f0012190.slice/crio-c2fc8af9bc70691879178cbb560ffdce8ea6509db00eed8e64b751deb8aa7196 WatchSource:0}: Error finding container c2fc8af9bc70691879178cbb560ffdce8ea6509db00eed8e64b751deb8aa7196: Status 404 returned error can't find the container with id c2fc8af9bc70691879178cbb560ffdce8ea6509db00eed8e64b751deb8aa7196 Nov 02 19:57:20.714333 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:20.714299 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" event={"ID":"b3fe0bf0-a9ee-4779-9343-c0f8f0012190","Type":"ContainerStarted","Data":"c2fc8af9bc70691879178cbb560ffdce8ea6509db00eed8e64b751deb8aa7196"} Nov 02 19:57:24.733957 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:24.733917 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" event={"ID":"b3fe0bf0-a9ee-4779-9343-c0f8f0012190","Type":"ContainerStarted","Data":"25010a82ef3cd8d6be35f4e3f59e70982afeb1e797693bcdee4736be7fcf1e69"} Nov 02 19:57:24.938047 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:24.937916 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vg5b6" podStartSLOduration=1.85335166 podStartE2EDuration="5.937899574s" totalImagesPullingTime="4.084547914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:24.890637644 +0000 UTC m=+287.501289736" watchObservedRunningTime="2026-11-02 19:57:24.937899574 +0000 UTC m=+287.548551665" Nov 02 19:57:25.898632 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:25.898598 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm"] Nov 02 19:57:25.901952 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:25.901935 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm" Nov 02 19:57:25.956425 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:25.956396 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lcn6t\"" Nov 02 19:57:26.002925 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.002899 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 19:57:26.003173 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.003154 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 19:57:26.005204 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.005178 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm"] Nov 02 19:57:26.050738 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.050691 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxvk\" (UniqueName: \"kubernetes.io/projected/e19d2022-b6cd-4cd2-90af-18082e986708-kube-api-access-tbxvk\") pod \"authorino-operator-7b7bb4bc77-5gnwm\" (UID: \"e19d2022-b6cd-4cd2-90af-18082e986708\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm" Nov 02 19:57:26.151321 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.151207 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxvk\" (UniqueName: \"kubernetes.io/projected/e19d2022-b6cd-4cd2-90af-18082e986708-kube-api-access-tbxvk\") pod \"authorino-operator-7b7bb4bc77-5gnwm\" (UID: \"e19d2022-b6cd-4cd2-90af-18082e986708\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm" Nov 02 19:57:26.204996 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.204970 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxvk\" (UniqueName: \"kubernetes.io/projected/e19d2022-b6cd-4cd2-90af-18082e986708-kube-api-access-tbxvk\") pod \"authorino-operator-7b7bb4bc77-5gnwm\" (UID: \"e19d2022-b6cd-4cd2-90af-18082e986708\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm" Nov 02 19:57:26.213792 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.213767 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm" Nov 02 19:57:26.579218 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.579184 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm"] Nov 02 19:57:26.582452 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:57:26.582420 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19d2022_b6cd_4cd2_90af_18082e986708.slice/crio-b4c2bf1c0571ef6ec4a0dcdc05f13e57a346d4f663d241efe39d0dfb0f2fd497 WatchSource:0}: Error finding container b4c2bf1c0571ef6ec4a0dcdc05f13e57a346d4f663d241efe39d0dfb0f2fd497: Status 404 returned error can't find the container with id b4c2bf1c0571ef6ec4a0dcdc05f13e57a346d4f663d241efe39d0dfb0f2fd497 Nov 02 19:57:26.743878 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.743841 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm" event={"ID":"e19d2022-b6cd-4cd2-90af-18082e986708","Type":"ContainerStarted","Data":"b4c2bf1c0571ef6ec4a0dcdc05f13e57a346d4f663d241efe39d0dfb0f2fd497"} Nov 02 19:57:26.772949 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.772915 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z869s"] Nov 02 19:57:26.777252 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.777237 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" Nov 02 19:57:26.830686 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.830608 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 19:57:26.830686 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.830625 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 19:57:26.836704 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.836685 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-cq4wb\"" Nov 02 19:57:26.846655 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.846633 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z869s"] Nov 02 19:57:26.857559 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.857535 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f4c5a3e-3827-4aae-ba70-506867f2afa0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z869s\" (UID: \"6f4c5a3e-3827-4aae-ba70-506867f2afa0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" Nov 02 19:57:26.857674 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.857653 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cph85\" (UniqueName: \"kubernetes.io/projected/6f4c5a3e-3827-4aae-ba70-506867f2afa0-kube-api-access-cph85\") pod \"cert-manager-webhook-5dbbbf75cc-z869s\" (UID: \"6f4c5a3e-3827-4aae-ba70-506867f2afa0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" Nov 02 19:57:26.958402 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.958364 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f4c5a3e-3827-4aae-ba70-506867f2afa0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z869s\" (UID: \"6f4c5a3e-3827-4aae-ba70-506867f2afa0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" Nov 02 19:57:26.958813 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:26.958428 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cph85\" (UniqueName: \"kubernetes.io/projected/6f4c5a3e-3827-4aae-ba70-506867f2afa0-kube-api-access-cph85\") pod \"cert-manager-webhook-5dbbbf75cc-z869s\" (UID: \"6f4c5a3e-3827-4aae-ba70-506867f2afa0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" Nov 02 19:57:27.012306 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:27.012262 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f4c5a3e-3827-4aae-ba70-506867f2afa0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z869s\" (UID: \"6f4c5a3e-3827-4aae-ba70-506867f2afa0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" Nov 02 19:57:27.012306 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:27.012301 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cph85\" (UniqueName: \"kubernetes.io/projected/6f4c5a3e-3827-4aae-ba70-506867f2afa0-kube-api-access-cph85\") pod \"cert-manager-webhook-5dbbbf75cc-z869s\" (UID: \"6f4c5a3e-3827-4aae-ba70-506867f2afa0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" Nov 02 19:57:27.088337 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:27.088252 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" Nov 02 19:57:27.453807 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:27.453415 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z869s"] Nov 02 19:57:27.456083 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:57:27.456026 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f4c5a3e_3827_4aae_ba70_506867f2afa0.slice/crio-c4d98a4d68e4d8e4d7f34884f3b5ad34e63f658f25c1602a38b0d171aef452a6 WatchSource:0}: Error finding container c4d98a4d68e4d8e4d7f34884f3b5ad34e63f658f25c1602a38b0d171aef452a6: Status 404 returned error can't find the container with id c4d98a4d68e4d8e4d7f34884f3b5ad34e63f658f25c1602a38b0d171aef452a6 Nov 02 19:57:27.749125 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:27.749042 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" event={"ID":"6f4c5a3e-3827-4aae-ba70-506867f2afa0","Type":"ContainerStarted","Data":"c4d98a4d68e4d8e4d7f34884f3b5ad34e63f658f25c1602a38b0d171aef452a6"} Nov 02 19:57:30.766576 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:30.766532 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm" event={"ID":"e19d2022-b6cd-4cd2-90af-18082e986708","Type":"ContainerStarted","Data":"0e4f82fff81aed4250494222aac6d8ced4663ec17468e78d31d053a5b5c718a5"} Nov 02 19:57:30.767061 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:30.766694 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm" Nov 02 19:57:30.874313 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:30.874230 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm" podStartSLOduration=2.379884053 podStartE2EDuration="5.874210065s" totalImagesPullingTime="3.494326012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:30.872657478 +0000 UTC m=+293.483309568" watchObservedRunningTime="2026-11-02 19:57:30.874210065 +0000 UTC m=+293.484862158" Nov 02 19:57:31.772174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:31.772132 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" event={"ID":"6f4c5a3e-3827-4aae-ba70-506867f2afa0","Type":"ContainerStarted","Data":"e82f40dd7638e2ae97c6c7c9d91a2378a193b8c4f385a1463b90aba20738f880"} Nov 02 19:57:31.878378 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:31.878322 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" podStartSLOduration=2.172013974 podStartE2EDuration="5.878306902s" totalImagesPullingTime="3.706292928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:31.877353912 +0000 UTC m=+294.488006002" watchObservedRunningTime="2026-11-02 19:57:31.878306902 +0000 UTC m=+294.488958984" Nov 02 19:57:32.776707 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:32.776679 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" Nov 02 19:57:33.151879 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.151844 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg"] Nov 02 19:57:33.155173 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.155157 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" Nov 02 19:57:33.208740 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.208707 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg"] Nov 02 19:57:33.253378 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.253343 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8mqrw\"" Nov 02 19:57:33.314497 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.314457 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c64fl\" (UniqueName: \"kubernetes.io/projected/c4131f5b-4e1d-46e3-99a1-6341264e1c9f-kube-api-access-c64fl\") pod \"cert-manager-cainjector-5c57bbd4fc-cxvcg\" (UID: \"c4131f5b-4e1d-46e3-99a1-6341264e1c9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" Nov 02 19:57:33.314693 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.314510 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4131f5b-4e1d-46e3-99a1-6341264e1c9f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-cxvcg\" (UID: \"c4131f5b-4e1d-46e3-99a1-6341264e1c9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" Nov 02 19:57:33.415901 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.415807 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64fl\" (UniqueName: \"kubernetes.io/projected/c4131f5b-4e1d-46e3-99a1-6341264e1c9f-kube-api-access-c64fl\") pod \"cert-manager-cainjector-5c57bbd4fc-cxvcg\" (UID: \"c4131f5b-4e1d-46e3-99a1-6341264e1c9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" Nov 02 19:57:33.415901 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.415869 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4131f5b-4e1d-46e3-99a1-6341264e1c9f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-cxvcg\" (UID: \"c4131f5b-4e1d-46e3-99a1-6341264e1c9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" Nov 02 19:57:33.469850 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.469809 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4131f5b-4e1d-46e3-99a1-6341264e1c9f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-cxvcg\" (UID: \"c4131f5b-4e1d-46e3-99a1-6341264e1c9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" Nov 02 19:57:33.469997 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.469921 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64fl\" (UniqueName: \"kubernetes.io/projected/c4131f5b-4e1d-46e3-99a1-6341264e1c9f-kube-api-access-c64fl\") pod \"cert-manager-cainjector-5c57bbd4fc-cxvcg\" (UID: \"c4131f5b-4e1d-46e3-99a1-6341264e1c9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" Nov 02 19:57:33.766566 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.766483 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" Nov 02 19:57:33.769176 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.769147 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl"] Nov 02 19:57:33.772445 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.772425 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:33.825257 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.825227 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 19:57:33.827802 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.827776 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 19:57:33.835694 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.835630 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 19:57:33.835996 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.835953 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 19:57:33.901487 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.901455 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl"] Nov 02 19:57:33.920742 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.919938 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:33.920742 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.920002 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-metrics-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:33.920742 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.920031 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-manifests\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:33.920742 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.920067 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:33.920742 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.920138 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-tmp\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:33.920742 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.920163 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w729z\" (UniqueName: \"kubernetes.io/projected/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-kube-api-access-w729z\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:33.932438 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.932388 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-km6nn\"" Nov 02 19:57:33.932438 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:33.932387 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 19:57:34.020733 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.020640 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-tmp\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.020733 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.020687 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w729z\" (UniqueName: \"kubernetes.io/projected/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-kube-api-access-w729z\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.020733 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.020727 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.021221 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.020747 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-metrics-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.021221 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.020765 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-manifests\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.021221 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.020794 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.021516 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.021428 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-manifests\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.023678 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.023647 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-metrics-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.023678 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.023660 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.023803 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.023676 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-tmp\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.023970 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.023947 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.075289 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.075255 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w729z\" (UniqueName: \"kubernetes.io/projected/b6005088-b1fa-4a8b-a9a7-4c6b758479ac-kube-api-access-w729z\") pod \"opendatahub-operator-controller-manager-7846bf97f7-zqkrl\" (UID: \"b6005088-b1fa-4a8b-a9a7-4c6b758479ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.102201 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.102169 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:34.134872 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.134849 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg"] Nov 02 19:57:34.136826 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:57:34.136795 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4131f5b_4e1d_46e3_99a1_6341264e1c9f.slice/crio-eb5cf46020acf57178355f5ff4611712cfe20251f6662bd6a1f56debb33fae37 WatchSource:0}: Error finding container eb5cf46020acf57178355f5ff4611712cfe20251f6662bd6a1f56debb33fae37: Status 404 returned error can't find the container with id eb5cf46020acf57178355f5ff4611712cfe20251f6662bd6a1f56debb33fae37 Nov 02 19:57:34.550981 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.550952 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl"] Nov 02 19:57:34.553100 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:57:34.553060 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6005088_b1fa_4a8b_a9a7_4c6b758479ac.slice/crio-d18aa626f15ff5c34eda667876b958a55fccfa5fe7dd195dd49db9e17fd9b4f7 WatchSource:0}: Error finding container d18aa626f15ff5c34eda667876b958a55fccfa5fe7dd195dd49db9e17fd9b4f7: Status 404 returned error can't find the container with id d18aa626f15ff5c34eda667876b958a55fccfa5fe7dd195dd49db9e17fd9b4f7 Nov 02 19:57:34.786792 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.786752 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" event={"ID":"c4131f5b-4e1d-46e3-99a1-6341264e1c9f","Type":"ContainerStarted","Data":"66ee5a2288416dd83f32575e8fd949d165113899d3e172ebdfdc4efce1f0d038"} Nov 02 19:57:34.786957 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.786810 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" event={"ID":"c4131f5b-4e1d-46e3-99a1-6341264e1c9f","Type":"ContainerStarted","Data":"eb5cf46020acf57178355f5ff4611712cfe20251f6662bd6a1f56debb33fae37"} Nov 02 19:57:34.788194 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.788161 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" event={"ID":"b6005088-b1fa-4a8b-a9a7-4c6b758479ac","Type":"ContainerStarted","Data":"d18aa626f15ff5c34eda667876b958a55fccfa5fe7dd195dd49db9e17fd9b4f7"} Nov 02 19:57:34.894837 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:34.894779 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cxvcg" podStartSLOduration=1.894765585 podStartE2EDuration="1.894765585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:34.893399041 +0000 UTC m=+297.504051133" watchObservedRunningTime="2026-11-02 19:57:34.894765585 +0000 UTC m=+297.505417676" Nov 02 19:57:38.784056 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:38.784021 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z869s" Nov 02 19:57:39.814343 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:39.814307 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" event={"ID":"b6005088-b1fa-4a8b-a9a7-4c6b758479ac","Type":"ContainerStarted","Data":"ee581eff5015b51f7c32200af12885f9eedfbcba653ceaacb4f830e8127306eb"} Nov 02 19:57:40.820600 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:40.820566 5399 generic.go:356] "Generic (PLEG): container finished" podID="b6005088-b1fa-4a8b-a9a7-4c6b758479ac" containerID="ee581eff5015b51f7c32200af12885f9eedfbcba653ceaacb4f830e8127306eb" exitCode=0 Nov 02 19:57:40.821000 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:40.820609 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" event={"ID":"b6005088-b1fa-4a8b-a9a7-4c6b758479ac","Type":"ContainerDied","Data":"ee581eff5015b51f7c32200af12885f9eedfbcba653ceaacb4f830e8127306eb"} Nov 02 19:57:40.821166 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:40.821148 5399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:57:41.774750 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:41.774717 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gnwm" Nov 02 19:57:41.827988 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:41.827953 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" event={"ID":"b6005088-b1fa-4a8b-a9a7-4c6b758479ac","Type":"ContainerStarted","Data":"d4dec638203cd2d01055d9b6a4b6d38acb33306973e8bed80c4d93c253d7332e"} Nov 02 19:57:41.828374 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:41.828041 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:42.055575 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:42.055460 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" podStartSLOduration=3.029343833 podStartE2EDuration="9.055439191s" totalImagesPullingTime="5.61246606s" totalInitContainerRuntime="413.629298ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:42.007832111 +0000 UTC m=+304.618484203" watchObservedRunningTime="2026-11-02 19:57:42.055439191 +0000 UTC m=+304.666091286" Nov 02 19:57:47.935919 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:47.935882 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r"] Nov 02 19:57:47.939836 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:47.939818 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:47.943774 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:47.943743 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19e0ccc9-2f20-4852-a57f-78f4c48b9920-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wbp8r\" (UID: \"19e0ccc9-2f20-4852-a57f-78f4c48b9920\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:47.943908 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:47.943846 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/19e0ccc9-2f20-4852-a57f-78f4c48b9920-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wbp8r\" (UID: \"19e0ccc9-2f20-4852-a57f-78f4c48b9920\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:47.943960 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:47.943908 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69ljm\" (UniqueName: \"kubernetes.io/projected/19e0ccc9-2f20-4852-a57f-78f4c48b9920-kube-api-access-69ljm\") pod \"kuadrant-console-plugin-66c4fb575d-wbp8r\" (UID: \"19e0ccc9-2f20-4852-a57f-78f4c48b9920\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:47.990631 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:47.990570 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 19:57:47.990954 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:47.990736 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 19:57:48.037074 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.037047 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8bwcw\"" Nov 02 19:57:48.038416 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.038396 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r"] Nov 02 19:57:48.044547 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.044528 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19e0ccc9-2f20-4852-a57f-78f4c48b9920-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wbp8r\" (UID: \"19e0ccc9-2f20-4852-a57f-78f4c48b9920\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:48.044652 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.044565 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/19e0ccc9-2f20-4852-a57f-78f4c48b9920-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wbp8r\" (UID: \"19e0ccc9-2f20-4852-a57f-78f4c48b9920\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:48.044652 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.044593 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69ljm\" (UniqueName: \"kubernetes.io/projected/19e0ccc9-2f20-4852-a57f-78f4c48b9920-kube-api-access-69ljm\") pod \"kuadrant-console-plugin-66c4fb575d-wbp8r\" (UID: \"19e0ccc9-2f20-4852-a57f-78f4c48b9920\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:48.044763 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:57:48.044689 5399 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 02 19:57:48.044763 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:57:48.044758 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19e0ccc9-2f20-4852-a57f-78f4c48b9920-plugin-serving-cert podName:19e0ccc9-2f20-4852-a57f-78f4c48b9920 nodeName:}" failed. No retries permitted until 2026-11-02 19:57:48.544740442 +0000 UTC m=+311.155392535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/19e0ccc9-2f20-4852-a57f-78f4c48b9920-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-wbp8r" (UID: "19e0ccc9-2f20-4852-a57f-78f4c48b9920") : secret "plugin-serving-cert" not found Nov 02 19:57:48.045209 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.045188 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/19e0ccc9-2f20-4852-a57f-78f4c48b9920-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-wbp8r\" (UID: \"19e0ccc9-2f20-4852-a57f-78f4c48b9920\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:48.098692 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.098659 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69ljm\" (UniqueName: \"kubernetes.io/projected/19e0ccc9-2f20-4852-a57f-78f4c48b9920-kube-api-access-69ljm\") pod \"kuadrant-console-plugin-66c4fb575d-wbp8r\" (UID: \"19e0ccc9-2f20-4852-a57f-78f4c48b9920\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:48.235060 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:57:48.234973 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:57:48.547027 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.546938 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/19e0ccc9-2f20-4852-a57f-78f4c48b9920-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wbp8r\" (UID: \"19e0ccc9-2f20-4852-a57f-78f4c48b9920\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:48.549406 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.549382 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/19e0ccc9-2f20-4852-a57f-78f4c48b9920-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-wbp8r\" (UID: \"19e0ccc9-2f20-4852-a57f-78f4c48b9920\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:48.552238 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.552207 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" Nov 02 19:57:48.915315 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:48.915165 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r"] Nov 02 19:57:48.917865 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:57:48.917829 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19e0ccc9_2f20_4852_a57f_78f4c48b9920.slice/crio-7e7a009ba810577a1df27eb36a378af3847dd87c6dca10bfa9ef451b4f243355 WatchSource:0}: Error finding container 7e7a009ba810577a1df27eb36a378af3847dd87c6dca10bfa9ef451b4f243355: Status 404 returned error can't find the container with id 7e7a009ba810577a1df27eb36a378af3847dd87c6dca10bfa9ef451b4f243355 Nov 02 19:57:49.864012 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:49.863977 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" event={"ID":"19e0ccc9-2f20-4852-a57f-78f4c48b9920","Type":"ContainerStarted","Data":"7e7a009ba810577a1df27eb36a378af3847dd87c6dca10bfa9ef451b4f243355"} Nov 02 19:57:52.835135 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:52.835103 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7846bf97f7-zqkrl" Nov 02 19:57:54.889613 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:54.889526 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" event={"ID":"19e0ccc9-2f20-4852-a57f-78f4c48b9920","Type":"ContainerStarted","Data":"c9230c3e62804d70a97bd3db280c01bd4b4c3f33290919c0f18ef10142229c17"} Nov 02 19:57:55.000393 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:57:55.000335 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-wbp8r" podStartSLOduration=2.304997101 podStartE2EDuration="8.000318904s" totalImagesPullingTime="5.695321803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:54.998448293 +0000 UTC m=+317.609100397" watchObservedRunningTime="2026-11-02 19:57:55.000318904 +0000 UTC m=+317.610970989" Nov 02 19:58:07.104335 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.104294 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67b88d44-mnwpp"] Nov 02 19:58:07.110754 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.110730 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.159215 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.159177 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 19:58:07.159412 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.159349 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 19:58:07.192055 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.192022 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab733748-d93c-4b7b-96ab-b4fb1f74ea1c-tls-certs\") pod \"kube-auth-proxy-67b88d44-mnwpp\" (UID: \"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.192055 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.192060 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab733748-d93c-4b7b-96ab-b4fb1f74ea1c-tmp\") pod \"kube-auth-proxy-67b88d44-mnwpp\" (UID: \"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.192258 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.192083 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkdj\" (UniqueName: \"kubernetes.io/projected/ab733748-d93c-4b7b-96ab-b4fb1f74ea1c-kube-api-access-hnkdj\") pod \"kube-auth-proxy-67b88d44-mnwpp\" (UID: \"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.205770 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.205727 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cjjfn\"" Nov 02 19:58:07.205950 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.205774 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 19:58:07.205950 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.205806 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 19:58:07.207260 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.207236 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67b88d44-mnwpp"] Nov 02 19:58:07.293327 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.293269 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab733748-d93c-4b7b-96ab-b4fb1f74ea1c-tls-certs\") pod \"kube-auth-proxy-67b88d44-mnwpp\" (UID: \"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.293327 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.293331 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab733748-d93c-4b7b-96ab-b4fb1f74ea1c-tmp\") pod \"kube-auth-proxy-67b88d44-mnwpp\" (UID: \"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.293585 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.293357 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkdj\" (UniqueName: \"kubernetes.io/projected/ab733748-d93c-4b7b-96ab-b4fb1f74ea1c-kube-api-access-hnkdj\") pod \"kube-auth-proxy-67b88d44-mnwpp\" (UID: \"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.295616 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.295587 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab733748-d93c-4b7b-96ab-b4fb1f74ea1c-tmp\") pod \"kube-auth-proxy-67b88d44-mnwpp\" (UID: \"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.295915 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.295891 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab733748-d93c-4b7b-96ab-b4fb1f74ea1c-tls-certs\") pod \"kube-auth-proxy-67b88d44-mnwpp\" (UID: \"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.346490 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.346455 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkdj\" (UniqueName: \"kubernetes.io/projected/ab733748-d93c-4b7b-96ab-b4fb1f74ea1c-kube-api-access-hnkdj\") pod \"kube-auth-proxy-67b88d44-mnwpp\" (UID: \"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c\") " pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.422071 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.421991 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" Nov 02 19:58:07.784684 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.784657 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67b88d44-mnwpp"] Nov 02 19:58:07.787078 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:58:07.787045 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab733748_d93c_4b7b_96ab_b4fb1f74ea1c.slice/crio-5df21ac578dc2758f8bd88e1f349bbdf45b4478792c8a883de2abb98b9e1931c WatchSource:0}: Error finding container 5df21ac578dc2758f8bd88e1f349bbdf45b4478792c8a883de2abb98b9e1931c: Status 404 returned error can't find the container with id 5df21ac578dc2758f8bd88e1f349bbdf45b4478792c8a883de2abb98b9e1931c Nov 02 19:58:07.947985 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:07.947949 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" event={"ID":"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c","Type":"ContainerStarted","Data":"5df21ac578dc2758f8bd88e1f349bbdf45b4478792c8a883de2abb98b9e1931c"} Nov 02 19:58:10.964379 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:10.964322 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" event={"ID":"ab733748-d93c-4b7b-96ab-b4fb1f74ea1c","Type":"ContainerStarted","Data":"2fd3040bbade75b42e5d06b9f05273df940e08b126bd7d80249bf8cb146afc10"} Nov 02 19:58:11.051934 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.051861 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c8nv4"] Nov 02 19:58:11.058593 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.058571 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c8nv4" Nov 02 19:58:11.071076 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.071030 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67b88d44-mnwpp" podStartSLOduration=1.117584686 podStartE2EDuration="4.071016379s" totalImagesPullingTime="2.953431693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:11.069419133 +0000 UTC m=+333.680071225" watchObservedRunningTime="2026-11-02 19:58:11.071016379 +0000 UTC m=+333.681668495" Nov 02 19:58:11.104964 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.104933 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c8nv4"] Nov 02 19:58:11.115687 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.115659 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5v79g\"" Nov 02 19:58:11.127697 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.127668 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knxtn\" (UniqueName: \"kubernetes.io/projected/995529c8-a1ec-4385-84f2-69617ebfbf1b-kube-api-access-knxtn\") pod \"authorino-57f7f8659f-c8nv4\" (UID: \"995529c8-a1ec-4385-84f2-69617ebfbf1b\") " pod="kuadrant-system/authorino-57f7f8659f-c8nv4" Nov 02 19:58:11.229438 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.229399 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knxtn\" (UniqueName: \"kubernetes.io/projected/995529c8-a1ec-4385-84f2-69617ebfbf1b-kube-api-access-knxtn\") pod \"authorino-57f7f8659f-c8nv4\" (UID: \"995529c8-a1ec-4385-84f2-69617ebfbf1b\") " pod="kuadrant-system/authorino-57f7f8659f-c8nv4" Nov 02 19:58:11.282321 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.282292 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knxtn\" (UniqueName: \"kubernetes.io/projected/995529c8-a1ec-4385-84f2-69617ebfbf1b-kube-api-access-knxtn\") pod \"authorino-57f7f8659f-c8nv4\" (UID: \"995529c8-a1ec-4385-84f2-69617ebfbf1b\") " pod="kuadrant-system/authorino-57f7f8659f-c8nv4" Nov 02 19:58:11.372124 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.372088 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c8nv4" Nov 02 19:58:11.736054 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.735964 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c8nv4"] Nov 02 19:58:11.739021 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:58:11.738986 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod995529c8_a1ec_4385_84f2_69617ebfbf1b.slice/crio-d9465d75630a1f43aaeea7487395110c0e27eaf3704b625001687ad787e86772 WatchSource:0}: Error finding container d9465d75630a1f43aaeea7487395110c0e27eaf3704b625001687ad787e86772: Status 404 returned error can't find the container with id d9465d75630a1f43aaeea7487395110c0e27eaf3704b625001687ad787e86772 Nov 02 19:58:11.970369 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:11.970332 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c8nv4" event={"ID":"995529c8-a1ec-4385-84f2-69617ebfbf1b","Type":"ContainerStarted","Data":"d9465d75630a1f43aaeea7487395110c0e27eaf3704b625001687ad787e86772"} Nov 02 19:58:12.779519 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:12.779484 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 19:58:12.785940 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:12.785920 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" Nov 02 19:58:12.834562 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:12.834533 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 19:58:12.842365 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:12.842337 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef52526-da07-44fd-af13-1fa9a89feddf-config-file\") pod \"limitador-limitador-7c9b87d574-vg6vn\" (UID: \"3ef52526-da07-44fd-af13-1fa9a89feddf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" Nov 02 19:58:12.842508 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:12.842370 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th6ht\" (UniqueName: \"kubernetes.io/projected/3ef52526-da07-44fd-af13-1fa9a89feddf-kube-api-access-th6ht\") pod \"limitador-limitador-7c9b87d574-vg6vn\" (UID: \"3ef52526-da07-44fd-af13-1fa9a89feddf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" Nov 02 19:58:12.879209 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:12.879179 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 19:58:12.943501 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:12.943464 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef52526-da07-44fd-af13-1fa9a89feddf-config-file\") pod \"limitador-limitador-7c9b87d574-vg6vn\" (UID: \"3ef52526-da07-44fd-af13-1fa9a89feddf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" Nov 02 19:58:12.943501 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:12.943502 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th6ht\" (UniqueName: \"kubernetes.io/projected/3ef52526-da07-44fd-af13-1fa9a89feddf-kube-api-access-th6ht\") pod \"limitador-limitador-7c9b87d574-vg6vn\" (UID: \"3ef52526-da07-44fd-af13-1fa9a89feddf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" Nov 02 19:58:12.944084 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:12.944062 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ef52526-da07-44fd-af13-1fa9a89feddf-config-file\") pod \"limitador-limitador-7c9b87d574-vg6vn\" (UID: \"3ef52526-da07-44fd-af13-1fa9a89feddf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" Nov 02 19:58:12.996720 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:12.996682 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th6ht\" (UniqueName: \"kubernetes.io/projected/3ef52526-da07-44fd-af13-1fa9a89feddf-kube-api-access-th6ht\") pod \"limitador-limitador-7c9b87d574-vg6vn\" (UID: \"3ef52526-da07-44fd-af13-1fa9a89feddf\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" Nov 02 19:58:13.084727 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:13.084697 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 19:58:13.098804 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:13.098782 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" Nov 02 19:58:13.458632 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:13.458608 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 19:58:13.460997 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:58:13.460969 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef52526_da07_44fd_af13_1fa9a89feddf.slice/crio-1bf1b45ca6b393b007bffda51f560e3de48342a1491dc77260c9878385c11a2b WatchSource:0}: Error finding container 1bf1b45ca6b393b007bffda51f560e3de48342a1491dc77260c9878385c11a2b: Status 404 returned error can't find the container with id 1bf1b45ca6b393b007bffda51f560e3de48342a1491dc77260c9878385c11a2b Nov 02 19:58:13.979590 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:13.979550 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" event={"ID":"3ef52526-da07-44fd-af13-1fa9a89feddf","Type":"ContainerStarted","Data":"1bf1b45ca6b393b007bffda51f560e3de48342a1491dc77260c9878385c11a2b"} Nov 02 19:58:15.990491 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:15.990453 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c8nv4" event={"ID":"995529c8-a1ec-4385-84f2-69617ebfbf1b","Type":"ContainerStarted","Data":"ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e"} Nov 02 19:58:15.992005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:15.991967 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" event={"ID":"3ef52526-da07-44fd-af13-1fa9a89feddf","Type":"ContainerStarted","Data":"c0289009e355e5ec801bdc5484c1b814a13a679ba4847430b862695a89e67311"} Nov 02 19:58:15.992135 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:15.992057 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" Nov 02 19:58:16.096374 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:16.096315 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-c8nv4" podStartSLOduration=1.533285832 podStartE2EDuration="5.096301644s" totalImagesPullingTime="3.563015812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:16.094025313 +0000 UTC m=+338.704677405" watchObservedRunningTime="2026-11-02 19:58:16.096301644 +0000 UTC m=+338.706953768" Nov 02 19:58:16.200159 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:16.200099 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" podStartSLOduration=2.197193357 podStartE2EDuration="4.200085368s" totalImagesPullingTime="2.002892011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:16.198504552 +0000 UTC m=+338.809156642" watchObservedRunningTime="2026-11-02 19:58:16.200085368 +0000 UTC m=+338.810737454" Nov 02 19:58:17.176991 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:17.176951 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c8nv4"] Nov 02 19:58:18.001374 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:18.001335 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-c8nv4" podUID="995529c8-a1ec-4385-84f2-69617ebfbf1b" containerName="authorino" containerID="cri-o://ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e" gracePeriod=30 Nov 02 19:58:18.238950 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:58:18.235976 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:58:18.243202 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:18.243180 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c8nv4" Nov 02 19:58:18.292509 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:18.292430 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995529c8-a1ec-4385-84f2-69617ebfbf1b-kube-api-access-knxtn\" (UniqueName: \"kubernetes.io/projected/995529c8-a1ec-4385-84f2-69617ebfbf1b-kube-api-access-knxtn\") pod \"995529c8-a1ec-4385-84f2-69617ebfbf1b\" (UID: \"995529c8-a1ec-4385-84f2-69617ebfbf1b\") " Nov 02 19:58:18.294584 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:18.294560 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995529c8-a1ec-4385-84f2-69617ebfbf1b-kube-api-access-knxtn" pod "995529c8-a1ec-4385-84f2-69617ebfbf1b" (UID: "995529c8-a1ec-4385-84f2-69617ebfbf1b"). InnerVolumeSpecName "kube-api-access-knxtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:58:18.393499 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:18.393460 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knxtn\" (UniqueName: \"kubernetes.io/projected/995529c8-a1ec-4385-84f2-69617ebfbf1b-kube-api-access-knxtn\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:58:19.006496 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:19.006463 5399 generic.go:356] "Generic (PLEG): container finished" podID="995529c8-a1ec-4385-84f2-69617ebfbf1b" containerID="ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e" exitCode=0 Nov 02 19:58:19.006676 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:19.006507 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c8nv4" Nov 02 19:58:19.006676 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:19.006559 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c8nv4" event={"ID":"995529c8-a1ec-4385-84f2-69617ebfbf1b","Type":"ContainerDied","Data":"ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e"} Nov 02 19:58:19.006676 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:19.006605 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c8nv4" event={"ID":"995529c8-a1ec-4385-84f2-69617ebfbf1b","Type":"ContainerDied","Data":"d9465d75630a1f43aaeea7487395110c0e27eaf3704b625001687ad787e86772"} Nov 02 19:58:19.006676 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:19.006622 5399 scope.go:122] "RemoveContainer" containerID="ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e" Nov 02 19:58:19.015160 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:19.015144 5399 scope.go:122] "RemoveContainer" containerID="ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e" Nov 02 19:58:19.015481 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:58:19.015461 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e\": container with ID starting with ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e not found: ID does not exist" containerID="ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e" Nov 02 19:58:19.015542 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:19.015492 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e"} err="failed to get container status \"ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e\": rpc error: code = NotFound desc = could not find container \"ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e\": container with ID starting with ad1e048e40c8f5af2e8e23622eae53a1110b850bcd27cb6a3b80cbd61c53792e not found: ID does not exist" Nov 02 19:58:19.162550 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:19.157640 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c8nv4"] Nov 02 19:58:19.203204 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:19.203161 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c8nv4"] Nov 02 19:58:19.506209 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:19.506170 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995529c8-a1ec-4385-84f2-69617ebfbf1b" path="/var/lib/kubelet/pods/995529c8-a1ec-4385-84f2-69617ebfbf1b/volumes" Nov 02 19:58:26.997799 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:26.997767 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-vg6vn" Nov 02 19:58:48.235304 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:58:48.235209 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:58:54.016185 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.016149 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-glsq7"] Nov 02 19:58:54.016705 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.016685 5399 cpu_manager.go:414] "containerMap: removing container" podUID="995529c8-a1ec-4385-84f2-69617ebfbf1b" containerName="authorino" Nov 02 19:58:54.016785 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.016709 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995529c8-a1ec-4385-84f2-69617ebfbf1b" containerName="authorino" Nov 02 19:58:54.016838 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.016814 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="995529c8-a1ec-4385-84f2-69617ebfbf1b" containerName="authorino" Nov 02 19:58:54.021002 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.020979 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.069937 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.069908 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 19:58:54.071609 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.071582 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-glsq7"] Nov 02 19:58:54.107184 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.107159 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-webhook-cert\") pod \"maas-controller-579cf7c6d-glsq7\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.107342 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.107205 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-metrics-certs\") pod \"maas-controller-579cf7c6d-glsq7\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.107395 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.107345 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt5bx\" (UniqueName: \"kubernetes.io/projected/001f0f44-2e81-404e-b31f-b4e99fd2f65e-kube-api-access-qt5bx\") pod \"maas-controller-579cf7c6d-glsq7\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.116686 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.116659 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 19:58:54.116686 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.116684 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 19:58:54.116851 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.116685 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 19:58:54.116851 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.116729 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 19:58:54.116851 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.116796 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h8zjd\"" Nov 02 19:58:54.208487 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.208457 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-webhook-cert\") pod \"maas-controller-579cf7c6d-glsq7\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.208643 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.208498 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-metrics-certs\") pod \"maas-controller-579cf7c6d-glsq7\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.208643 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.208551 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt5bx\" (UniqueName: \"kubernetes.io/projected/001f0f44-2e81-404e-b31f-b4e99fd2f65e-kube-api-access-qt5bx\") pod \"maas-controller-579cf7c6d-glsq7\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.208718 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:58:54.208666 5399 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Nov 02 19:58:54.208750 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:58:54.208736 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-metrics-certs podName:001f0f44-2e81-404e-b31f-b4e99fd2f65e nodeName:}" failed. No retries permitted until 2026-11-02 19:58:54.708720481 +0000 UTC m=+377.319372563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-metrics-certs") pod "maas-controller-579cf7c6d-glsq7" (UID: "001f0f44-2e81-404e-b31f-b4e99fd2f65e") : secret "maas-controller-metrics-tls" not found Nov 02 19:58:54.211059 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.211036 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-webhook-cert\") pod \"maas-controller-579cf7c6d-glsq7\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.261649 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.261623 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt5bx\" (UniqueName: \"kubernetes.io/projected/001f0f44-2e81-404e-b31f-b4e99fd2f65e-kube-api-access-qt5bx\") pod \"maas-controller-579cf7c6d-glsq7\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.397112 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.397075 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-zxn5f"] Nov 02 19:58:54.404069 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.404039 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.410190 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.410161 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-metrics-certs\") pod \"maas-controller-5ff67f6f5f-zxn5f\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.410324 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.410257 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9kq\" (UniqueName: \"kubernetes.io/projected/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-kube-api-access-sc9kq\") pod \"maas-controller-5ff67f6f5f-zxn5f\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.410396 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.410361 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-webhook-cert\") pod \"maas-controller-5ff67f6f5f-zxn5f\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.454375 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.454338 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-zxn5f"] Nov 02 19:58:54.510999 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.510967 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-webhook-cert\") pod \"maas-controller-5ff67f6f5f-zxn5f\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.511172 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.511029 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-metrics-certs\") pod \"maas-controller-5ff67f6f5f-zxn5f\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.511237 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.511203 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9kq\" (UniqueName: \"kubernetes.io/projected/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-kube-api-access-sc9kq\") pod \"maas-controller-5ff67f6f5f-zxn5f\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.513458 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.513431 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-metrics-certs\") pod \"maas-controller-5ff67f6f5f-zxn5f\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.513566 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.513490 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-webhook-cert\") pod \"maas-controller-5ff67f6f5f-zxn5f\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.563782 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.563752 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9kq\" (UniqueName: \"kubernetes.io/projected/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-kube-api-access-sc9kq\") pod \"maas-controller-5ff67f6f5f-zxn5f\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.712897 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.712813 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-metrics-certs\") pod \"maas-controller-579cf7c6d-glsq7\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.715123 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.715097 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-metrics-certs\") pod \"maas-controller-579cf7c6d-glsq7\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:54.717981 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.717963 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:54.935134 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:54.935101 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:55.078551 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:55.078525 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-zxn5f"] Nov 02 19:58:55.080362 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:58:55.080338 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbf52da_05a4_42fb_b6c2_c3ebc3ec2036.slice/crio-24ca67f2ade289c449c9ff5267de8f310d943ab5d7488222eac10ba06e307792 WatchSource:0}: Error finding container 24ca67f2ade289c449c9ff5267de8f310d943ab5d7488222eac10ba06e307792: Status 404 returned error can't find the container with id 24ca67f2ade289c449c9ff5267de8f310d943ab5d7488222eac10ba06e307792 Nov 02 19:58:55.169091 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:55.169052 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" event={"ID":"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036","Type":"ContainerStarted","Data":"24ca67f2ade289c449c9ff5267de8f310d943ab5d7488222eac10ba06e307792"} Nov 02 19:58:55.295203 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:55.295177 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-glsq7"] Nov 02 19:58:55.297507 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:58:55.297477 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001f0f44_2e81_404e_b31f_b4e99fd2f65e.slice/crio-1e28a9f475add4c9e3c59d0c4a150cd31d27341748a1cf190e48cbd1a6db4be2 WatchSource:0}: Error finding container 1e28a9f475add4c9e3c59d0c4a150cd31d27341748a1cf190e48cbd1a6db4be2: Status 404 returned error can't find the container with id 1e28a9f475add4c9e3c59d0c4a150cd31d27341748a1cf190e48cbd1a6db4be2 Nov 02 19:58:56.176518 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:56.176478 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-glsq7" event={"ID":"001f0f44-2e81-404e-b31f-b4e99fd2f65e","Type":"ContainerStarted","Data":"1e28a9f475add4c9e3c59d0c4a150cd31d27341748a1cf190e48cbd1a6db4be2"} Nov 02 19:58:59.195799 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:59.195767 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" event={"ID":"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036","Type":"ContainerStarted","Data":"e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c"} Nov 02 19:58:59.196217 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:59.195829 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:58:59.197294 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:59.197253 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-glsq7" event={"ID":"001f0f44-2e81-404e-b31f-b4e99fd2f65e","Type":"ContainerStarted","Data":"a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb"} Nov 02 19:58:59.197384 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:59.197315 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:58:59.303596 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:59.303541 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" podStartSLOduration=1.618880439 podStartE2EDuration="5.303525965s" totalImagesPullingTime="3.684645526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:59.302168213 +0000 UTC m=+381.912820303" watchObservedRunningTime="2026-11-02 19:58:59.303525965 +0000 UTC m=+381.914178056" Nov 02 19:58:59.411906 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:58:59.411807 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-glsq7" podStartSLOduration=3.199895438 podStartE2EDuration="6.411792601s" totalImagesPullingTime="3.211897163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:59.409575285 +0000 UTC m=+382.020227373" watchObservedRunningTime="2026-11-02 19:58:59.411792601 +0000 UTC m=+382.022444692" Nov 02 19:59:01.633999 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.633962 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4"] Nov 02 19:59:01.637515 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.637500 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:01.675417 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.675380 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dxx\" (UniqueName: \"kubernetes.io/projected/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-kube-api-access-z8dxx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:01.675581 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.675429 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:01.675581 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.675515 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:01.675581 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.675577 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:01.690790 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.690761 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4"] Nov 02 19:59:01.737526 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.737489 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 19:59:01.737526 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.737514 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 19:59:01.737772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.737579 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-tswkl\"" Nov 02 19:59:01.737772 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.737582 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 19:59:01.776961 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.776928 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dxx\" (UniqueName: \"kubernetes.io/projected/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-kube-api-access-z8dxx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:01.777131 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.776969 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:01.777131 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:59:01.777097 5399 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 19:59:01.777232 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:59:01.777152 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-proxy-source podName:b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01 nodeName:}" failed. No retries permitted until 2026-11-02 19:59:02.277136357 +0000 UTC m=+384.887788439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" (UID: "b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01") : configmap "usage-tenancy-proxy-source" not found Nov 02 19:59:01.777232 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.777165 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:01.777232 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.777202 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:01.777419 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:59:01.777291 5399 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 02 19:59:01.777419 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:59:01.777323 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-tls-cert podName:b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01 nodeName:}" failed. No retries permitted until 2026-11-02 19:59:02.277315445 +0000 UTC m=+384.887967514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" (UID: "b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01") : secret "usage-logs-tenancy-proxy-tls" not found Nov 02 19:59:01.777838 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.777815 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:01.831570 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:01.831537 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dxx\" (UniqueName: \"kubernetes.io/projected/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-kube-api-access-z8dxx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:02.281601 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:02.281569 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:02.281776 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:02.281632 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:02.282417 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:02.282387 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:02.283918 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:02.283900 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4k4s4\" (UID: \"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:02.549519 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:02.549414 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:02.909825 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:02.909797 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4"] Nov 02 19:59:02.912437 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:59:02.912408 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e7cfc8_fd01_4be3_a2e4_f3c8c0bf1f01.slice/crio-b0220ff3e1ed543abef7c7e1e2fac3589f69f835fa14052f6754c226cce4e7f9 WatchSource:0}: Error finding container b0220ff3e1ed543abef7c7e1e2fac3589f69f835fa14052f6754c226cce4e7f9: Status 404 returned error can't find the container with id b0220ff3e1ed543abef7c7e1e2fac3589f69f835fa14052f6754c226cce4e7f9 Nov 02 19:59:03.219454 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:03.219364 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" event={"ID":"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01","Type":"ContainerStarted","Data":"b0220ff3e1ed543abef7c7e1e2fac3589f69f835fa14052f6754c226cce4e7f9"} Nov 02 19:59:10.205685 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.205648 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:59:10.206942 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.206914 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 19:59:10.464771 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.464687 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-glsq7"] Nov 02 19:59:10.464984 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.464932 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-glsq7" podUID="001f0f44-2e81-404e-b31f-b4e99fd2f65e" containerName="manager" containerID="cri-o://a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb" gracePeriod=30 Nov 02 19:59:10.740549 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.740523 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:59:10.865327 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.865296 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001f0f44-2e81-404e-b31f-b4e99fd2f65e-kube-api-access-qt5bx\" (UniqueName: \"kubernetes.io/projected/001f0f44-2e81-404e-b31f-b4e99fd2f65e-kube-api-access-qt5bx\") pod \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " Nov 02 19:59:10.865520 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.865340 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-metrics-certs\") pod \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " Nov 02 19:59:10.865520 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.865389 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-webhook-cert\") pod \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\" (UID: \"001f0f44-2e81-404e-b31f-b4e99fd2f65e\") " Nov 02 19:59:10.869161 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.869122 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-webhook-cert" pod "001f0f44-2e81-404e-b31f-b4e99fd2f65e" (UID: "001f0f44-2e81-404e-b31f-b4e99fd2f65e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:59:10.869536 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.869509 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001f0f44-2e81-404e-b31f-b4e99fd2f65e-kube-api-access-qt5bx" pod "001f0f44-2e81-404e-b31f-b4e99fd2f65e" (UID: "001f0f44-2e81-404e-b31f-b4e99fd2f65e"). InnerVolumeSpecName "kube-api-access-qt5bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:59:10.870565 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.870539 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-metrics-certs" pod "001f0f44-2e81-404e-b31f-b4e99fd2f65e" (UID: "001f0f44-2e81-404e-b31f-b4e99fd2f65e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:59:10.966648 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.966610 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt5bx\" (UniqueName: \"kubernetes.io/projected/001f0f44-2e81-404e-b31f-b4e99fd2f65e-kube-api-access-qt5bx\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:10.966648 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.966643 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:10.966648 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:10.966653 5399 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/001f0f44-2e81-404e-b31f-b4e99fd2f65e-webhook-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:11.264929 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:11.264889 5399 generic.go:356] "Generic (PLEG): container finished" podID="001f0f44-2e81-404e-b31f-b4e99fd2f65e" containerID="a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb" exitCode=0 Nov 02 19:59:11.265438 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:11.264936 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-glsq7" Nov 02 19:59:11.265438 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:11.264977 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-glsq7" event={"ID":"001f0f44-2e81-404e-b31f-b4e99fd2f65e","Type":"ContainerDied","Data":"a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb"} Nov 02 19:59:11.265438 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:11.265036 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-glsq7" event={"ID":"001f0f44-2e81-404e-b31f-b4e99fd2f65e","Type":"ContainerDied","Data":"1e28a9f475add4c9e3c59d0c4a150cd31d27341748a1cf190e48cbd1a6db4be2"} Nov 02 19:59:11.265438 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:11.265060 5399 scope.go:122] "RemoveContainer" containerID="a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb" Nov 02 19:59:11.275691 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:11.275666 5399 scope.go:122] "RemoveContainer" containerID="a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb" Nov 02 19:59:11.276015 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:59:11.275994 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb\": container with ID starting with a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb not found: ID does not exist" containerID="a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb" Nov 02 19:59:11.276100 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:11.276024 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb"} err="failed to get container status \"a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb\": rpc error: code = NotFound desc = could not find container \"a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb\": container with ID starting with a1f2ddfd4888a7340ea7ac230a19800ab2eb030b26492e50a62fbcbed50b0feb not found: ID does not exist" Nov 02 19:59:11.423675 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:11.423635 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-glsq7"] Nov 02 19:59:11.427400 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:11.427346 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-glsq7"] Nov 02 19:59:11.505981 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:11.505946 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="001f0f44-2e81-404e-b31f-b4e99fd2f65e" path="/var/lib/kubelet/pods/001f0f44-2e81-404e-b31f-b4e99fd2f65e/volumes" Nov 02 19:59:16.300901 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:16.300863 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" event={"ID":"b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01","Type":"ContainerStarted","Data":"80b97ffd6e0ae1b04619cd1836a440b18cbfd63ee1fa09d0fd040c7a43edde31"} Nov 02 19:59:16.301298 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:16.300974 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:16.417258 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:16.417151 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" podStartSLOduration=2.190606235 podStartE2EDuration="15.417135102s" totalImagesPullingTime="13.226528867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:16.414670582 +0000 UTC m=+399.025322673" watchObservedRunningTime="2026-11-02 19:59:16.417135102 +0000 UTC m=+399.027787190" Nov 02 19:59:18.234583 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:59:18.234537 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 19:59:25.656555 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.656521 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lmtgk"] Nov 02 19:59:25.657034 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.656860 5399 cpu_manager.go:414] "containerMap: removing container" podUID="001f0f44-2e81-404e-b31f-b4e99fd2f65e" containerName="manager" Nov 02 19:59:25.657034 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.656871 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001f0f44-2e81-404e-b31f-b4e99fd2f65e" containerName="manager" Nov 02 19:59:25.657034 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.656939 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="001f0f44-2e81-404e-b31f-b4e99fd2f65e" containerName="manager" Nov 02 19:59:25.663383 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.663289 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" Nov 02 19:59:25.758516 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.758474 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 19:59:25.758516 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.758475 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2swlr\"" Nov 02 19:59:25.760356 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.760331 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lmtgk"] Nov 02 19:59:25.771367 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.771341 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp"] Nov 02 19:59:25.775375 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.775354 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 19:59:25.798702 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.798675 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d458\" (UniqueName: \"kubernetes.io/projected/4564bc57-50c6-41f8-9fb6-4c414641e45d-kube-api-access-4d458\") pod \"kserve-controller-manager-776cc66d47-lmtgk\" (UID: \"4564bc57-50c6-41f8-9fb6-4c414641e45d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" Nov 02 19:59:25.798826 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.798735 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4564bc57-50c6-41f8-9fb6-4c414641e45d-cert\") pod \"kserve-controller-manager-776cc66d47-lmtgk\" (UID: \"4564bc57-50c6-41f8-9fb6-4c414641e45d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" Nov 02 19:59:25.824900 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.824856 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nddwp\"" Nov 02 19:59:25.825436 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.825412 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 19:59:25.826682 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.826598 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp"] Nov 02 19:59:25.885230 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.885194 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vrfc2"] Nov 02 19:59:25.889283 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.889255 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" Nov 02 19:59:25.899494 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.899471 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d458\" (UniqueName: \"kubernetes.io/projected/4564bc57-50c6-41f8-9fb6-4c414641e45d-kube-api-access-4d458\") pod \"kserve-controller-manager-776cc66d47-lmtgk\" (UID: \"4564bc57-50c6-41f8-9fb6-4c414641e45d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" Nov 02 19:59:25.899600 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.899514 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6tl\" (UniqueName: \"kubernetes.io/projected/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-kube-api-access-gt6tl\") pod \"llmisvc-controller-manager-869d5dcc5d-2wbsp\" (UID: \"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 19:59:25.899600 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.899564 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2wbsp\" (UID: \"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 19:59:25.899701 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.899614 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4564bc57-50c6-41f8-9fb6-4c414641e45d-cert\") pod \"kserve-controller-manager-776cc66d47-lmtgk\" (UID: \"4564bc57-50c6-41f8-9fb6-4c414641e45d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" Nov 02 19:59:25.902036 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.902014 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4564bc57-50c6-41f8-9fb6-4c414641e45d-cert\") pod \"kserve-controller-manager-776cc66d47-lmtgk\" (UID: \"4564bc57-50c6-41f8-9fb6-4c414641e45d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" Nov 02 19:59:25.939513 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.939444 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-nk6dx\"" Nov 02 19:59:25.939513 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.939462 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 19:59:25.941092 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.941068 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vrfc2"] Nov 02 19:59:25.952347 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.952324 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d458\" (UniqueName: \"kubernetes.io/projected/4564bc57-50c6-41f8-9fb6-4c414641e45d-kube-api-access-4d458\") pod \"kserve-controller-manager-776cc66d47-lmtgk\" (UID: \"4564bc57-50c6-41f8-9fb6-4c414641e45d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" Nov 02 19:59:25.977395 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:25.977373 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" Nov 02 19:59:26.001046 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.001013 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6tl\" (UniqueName: \"kubernetes.io/projected/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-kube-api-access-gt6tl\") pod \"llmisvc-controller-manager-869d5dcc5d-2wbsp\" (UID: \"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 19:59:26.001191 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.001068 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2wbsp\" (UID: \"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 19:59:26.001191 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.001126 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa48662-147c-4804-abf3-737dc68d42d3-tls-certs\") pod \"model-serving-api-75bd89fdfc-vrfc2\" (UID: \"7fa48662-147c-4804-abf3-737dc68d42d3\") " pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" Nov 02 19:59:26.001191 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.001171 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2nv7\" (UniqueName: \"kubernetes.io/projected/7fa48662-147c-4804-abf3-737dc68d42d3-kube-api-access-m2nv7\") pod \"model-serving-api-75bd89fdfc-vrfc2\" (UID: \"7fa48662-147c-4804-abf3-737dc68d42d3\") " pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" Nov 02 19:59:26.003576 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.003549 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2wbsp\" (UID: \"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 19:59:26.054652 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.054625 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6tl\" (UniqueName: \"kubernetes.io/projected/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-kube-api-access-gt6tl\") pod \"llmisvc-controller-manager-869d5dcc5d-2wbsp\" (UID: \"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 19:59:26.088404 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.088370 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 19:59:26.102367 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.102337 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2nv7\" (UniqueName: \"kubernetes.io/projected/7fa48662-147c-4804-abf3-737dc68d42d3-kube-api-access-m2nv7\") pod \"model-serving-api-75bd89fdfc-vrfc2\" (UID: \"7fa48662-147c-4804-abf3-737dc68d42d3\") " pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" Nov 02 19:59:26.102487 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.102464 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa48662-147c-4804-abf3-737dc68d42d3-tls-certs\") pod \"model-serving-api-75bd89fdfc-vrfc2\" (UID: \"7fa48662-147c-4804-abf3-737dc68d42d3\") " pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" Nov 02 19:59:26.105062 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.105033 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa48662-147c-4804-abf3-737dc68d42d3-tls-certs\") pod \"model-serving-api-75bd89fdfc-vrfc2\" (UID: \"7fa48662-147c-4804-abf3-737dc68d42d3\") " pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" Nov 02 19:59:26.155821 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.155789 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2nv7\" (UniqueName: \"kubernetes.io/projected/7fa48662-147c-4804-abf3-737dc68d42d3-kube-api-access-m2nv7\") pod \"model-serving-api-75bd89fdfc-vrfc2\" (UID: \"7fa48662-147c-4804-abf3-737dc68d42d3\") " pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" Nov 02 19:59:26.201849 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.201767 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" Nov 02 19:59:26.342154 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.342118 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-lmtgk"] Nov 02 19:59:26.456286 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.456138 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp"] Nov 02 19:59:26.458601 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:59:26.458559 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0e4ecfc2_2e24_4bc2_bf0a_c4056cbac714.slice/crio-1f9fe0dd0e2692c0534ca761c13bf803aa58b1b573e0a808f516562f03b35021 WatchSource:0}: Error finding container 1f9fe0dd0e2692c0534ca761c13bf803aa58b1b573e0a808f516562f03b35021: Status 404 returned error can't find the container with id 1f9fe0dd0e2692c0534ca761c13bf803aa58b1b573e0a808f516562f03b35021 Nov 02 19:59:26.565009 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:26.564983 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vrfc2"] Nov 02 19:59:26.567081 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:59:26.567047 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa48662_147c_4804_abf3_737dc68d42d3.slice/crio-6ed80d4ce2bf692b0ea0f3d12bc8d4fca5dd22a856cd96b8357f40ff946509e5 WatchSource:0}: Error finding container 6ed80d4ce2bf692b0ea0f3d12bc8d4fca5dd22a856cd96b8357f40ff946509e5: Status 404 returned error can't find the container with id 6ed80d4ce2bf692b0ea0f3d12bc8d4fca5dd22a856cd96b8357f40ff946509e5 Nov 02 19:59:27.311860 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:27.311828 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4k4s4" Nov 02 19:59:27.354576 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:27.354533 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" event={"ID":"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714","Type":"ContainerStarted","Data":"1f9fe0dd0e2692c0534ca761c13bf803aa58b1b573e0a808f516562f03b35021"} Nov 02 19:59:27.355835 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:27.355806 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" event={"ID":"4564bc57-50c6-41f8-9fb6-4c414641e45d","Type":"ContainerStarted","Data":"aefb45174d3c5c1187ccdf186c4de142126646c8acf46b4d14563ab74dcedc5c"} Nov 02 19:59:27.356961 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:27.356936 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" event={"ID":"7fa48662-147c-4804-abf3-737dc68d42d3","Type":"ContainerStarted","Data":"6ed80d4ce2bf692b0ea0f3d12bc8d4fca5dd22a856cd96b8357f40ff946509e5"} Nov 02 19:59:30.378217 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:30.378179 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" event={"ID":"7fa48662-147c-4804-abf3-737dc68d42d3","Type":"ContainerStarted","Data":"a1cac2124672373701f83644eeae44664f743cb0b465844fc8a74e5f3bf1797d"} Nov 02 19:59:30.378699 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:30.378318 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" Nov 02 19:59:30.485459 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:30.485409 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" podStartSLOduration=2.182564609 podStartE2EDuration="5.485393848s" totalImagesPullingTime="3.302829239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:30.483203424 +0000 UTC m=+413.093855517" watchObservedRunningTime="2026-11-02 19:59:30.485393848 +0000 UTC m=+413.096045939" Nov 02 19:59:31.385197 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:31.385152 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" event={"ID":"4564bc57-50c6-41f8-9fb6-4c414641e45d","Type":"ContainerStarted","Data":"b9fb30be72d51783b9cac5b34ab84649fdee34820904c507a3e59a6af699abdc"} Nov 02 19:59:31.385762 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:31.385229 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" Nov 02 19:59:31.386763 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:31.386737 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" event={"ID":"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714","Type":"ContainerStarted","Data":"a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90"} Nov 02 19:59:31.386864 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:31.386849 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 19:59:31.492102 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:31.492052 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" podStartSLOduration=2.183128856 podStartE2EDuration="6.492036991s" totalImagesPullingTime="4.308908135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:31.490636397 +0000 UTC m=+414.101288500" watchObservedRunningTime="2026-11-02 19:59:31.492036991 +0000 UTC m=+414.102689082" Nov 02 19:59:31.609331 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:31.609258 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" podStartSLOduration=2.607813828 podStartE2EDuration="6.609242644s" totalImagesPullingTime="4.001428816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:31.608606359 +0000 UTC m=+414.219258451" watchObservedRunningTime="2026-11-02 19:59:31.609242644 +0000 UTC m=+414.219894734" Nov 02 19:59:36.187587 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:36.187548 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 19:59:41.391160 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:41.391124 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-vrfc2" Nov 02 19:59:41.711545 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:41.711465 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lpwdx"] Nov 02 19:59:41.728696 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:41.728666 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" Nov 02 19:59:41.766907 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:41.766874 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lpwdx"] Nov 02 19:59:41.777259 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:41.777227 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5v79g\"" Nov 02 19:59:41.844708 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:41.844673 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz8bz\" (UniqueName: \"kubernetes.io/projected/1edb0b19-8c3f-486e-b0d8-05f471d57ff0-kube-api-access-cz8bz\") pod \"authorino-6f6944bcdd-lpwdx\" (UID: \"1edb0b19-8c3f-486e-b0d8-05f471d57ff0\") " pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" Nov 02 19:59:41.946194 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:41.946154 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz8bz\" (UniqueName: \"kubernetes.io/projected/1edb0b19-8c3f-486e-b0d8-05f471d57ff0-kube-api-access-cz8bz\") pod \"authorino-6f6944bcdd-lpwdx\" (UID: \"1edb0b19-8c3f-486e-b0d8-05f471d57ff0\") " pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" Nov 02 19:59:42.001718 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.001655 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz8bz\" (UniqueName: \"kubernetes.io/projected/1edb0b19-8c3f-486e-b0d8-05f471d57ff0-kube-api-access-cz8bz\") pod \"authorino-6f6944bcdd-lpwdx\" (UID: \"1edb0b19-8c3f-486e-b0d8-05f471d57ff0\") " pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" Nov 02 19:59:42.040123 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.040093 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" Nov 02 19:59:42.050398 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.050366 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lpwdx"] Nov 02 19:59:42.067513 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.067482 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-586f56887b-8xt64"] Nov 02 19:59:42.071254 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.071233 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586f56887b-8xt64" Nov 02 19:59:42.123719 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.123679 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-586f56887b-8xt64"] Nov 02 19:59:42.249203 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.249167 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mcp\" (UniqueName: \"kubernetes.io/projected/433b226f-827b-40ef-ba68-2266e5ffca56-kube-api-access-z2mcp\") pod \"authorino-586f56887b-8xt64\" (UID: \"433b226f-827b-40ef-ba68-2266e5ffca56\") " pod="kuadrant-system/authorino-586f56887b-8xt64" Nov 02 19:59:42.350360 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.350319 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mcp\" (UniqueName: \"kubernetes.io/projected/433b226f-827b-40ef-ba68-2266e5ffca56-kube-api-access-z2mcp\") pod \"authorino-586f56887b-8xt64\" (UID: \"433b226f-827b-40ef-ba68-2266e5ffca56\") " pod="kuadrant-system/authorino-586f56887b-8xt64" Nov 02 19:59:42.401435 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.401410 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lpwdx"] Nov 02 19:59:42.403160 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:59:42.403130 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1edb0b19_8c3f_486e_b0d8_05f471d57ff0.slice/crio-eec0994eeb7384549e96bf79de6a9e5e0b257ef47e8ebc0873548e8b183ee3c7 WatchSource:0}: Error finding container eec0994eeb7384549e96bf79de6a9e5e0b257ef47e8ebc0873548e8b183ee3c7: Status 404 returned error can't find the container with id eec0994eeb7384549e96bf79de6a9e5e0b257ef47e8ebc0873548e8b183ee3c7 Nov 02 19:59:42.403836 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.403812 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mcp\" (UniqueName: \"kubernetes.io/projected/433b226f-827b-40ef-ba68-2266e5ffca56-kube-api-access-z2mcp\") pod \"authorino-586f56887b-8xt64\" (UID: \"433b226f-827b-40ef-ba68-2266e5ffca56\") " pod="kuadrant-system/authorino-586f56887b-8xt64" Nov 02 19:59:42.438676 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.438647 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" event={"ID":"1edb0b19-8c3f-486e-b0d8-05f471d57ff0","Type":"ContainerStarted","Data":"eec0994eeb7384549e96bf79de6a9e5e0b257ef47e8ebc0873548e8b183ee3c7"} Nov 02 19:59:42.514756 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.514718 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-586f56887b-8xt64"] Nov 02 19:59:42.514940 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.514929 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586f56887b-8xt64" Nov 02 19:59:42.540546 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.540514 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f9ff88c54-g5rmj"] Nov 02 19:59:42.545174 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.545142 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" Nov 02 19:59:42.595476 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.595440 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 19:59:42.596164 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.596139 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f9ff88c54-g5rmj"] Nov 02 19:59:42.652726 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.652640 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8p2v\" (UniqueName: \"kubernetes.io/projected/c4bd8a7e-f968-44f5-a052-d37741cf404e-kube-api-access-l8p2v\") pod \"authorino-5f9ff88c54-g5rmj\" (UID: \"c4bd8a7e-f968-44f5-a052-d37741cf404e\") " pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" Nov 02 19:59:42.652726 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.652696 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4bd8a7e-f968-44f5-a052-d37741cf404e-tls-cert\") pod \"authorino-5f9ff88c54-g5rmj\" (UID: \"c4bd8a7e-f968-44f5-a052-d37741cf404e\") " pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" Nov 02 19:59:42.753817 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.753774 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8p2v\" (UniqueName: \"kubernetes.io/projected/c4bd8a7e-f968-44f5-a052-d37741cf404e-kube-api-access-l8p2v\") pod \"authorino-5f9ff88c54-g5rmj\" (UID: \"c4bd8a7e-f968-44f5-a052-d37741cf404e\") " pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" Nov 02 19:59:42.753985 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.753834 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4bd8a7e-f968-44f5-a052-d37741cf404e-tls-cert\") pod \"authorino-5f9ff88c54-g5rmj\" (UID: \"c4bd8a7e-f968-44f5-a052-d37741cf404e\") " pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" Nov 02 19:59:42.756249 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.756229 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4bd8a7e-f968-44f5-a052-d37741cf404e-tls-cert\") pod \"authorino-5f9ff88c54-g5rmj\" (UID: \"c4bd8a7e-f968-44f5-a052-d37741cf404e\") " pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" Nov 02 19:59:42.806961 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.806929 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8p2v\" (UniqueName: \"kubernetes.io/projected/c4bd8a7e-f968-44f5-a052-d37741cf404e-kube-api-access-l8p2v\") pod \"authorino-5f9ff88c54-g5rmj\" (UID: \"c4bd8a7e-f968-44f5-a052-d37741cf404e\") " pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" Nov 02 19:59:42.860818 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.860781 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" Nov 02 19:59:42.879560 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:42.879536 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-586f56887b-8xt64"] Nov 02 19:59:42.881494 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:59:42.881469 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod433b226f_827b_40ef_ba68_2266e5ffca56.slice/crio-fc0eea14d14321abb3c5c51c5f03e462d31bc238029b0a1307089cb3746e3f43 WatchSource:0}: Error finding container fc0eea14d14321abb3c5c51c5f03e462d31bc238029b0a1307089cb3746e3f43: Status 404 returned error can't find the container with id fc0eea14d14321abb3c5c51c5f03e462d31bc238029b0a1307089cb3746e3f43 Nov 02 19:59:43.029222 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:43.029190 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f9ff88c54-g5rmj"] Nov 02 19:59:43.220791 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:43.220764 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f9ff88c54-g5rmj"] Nov 02 19:59:43.222808 ip-10-0-130-111 kubenswrapper[5399]: W1102 19:59:43.222781 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4bd8a7e_f968_44f5_a052_d37741cf404e.slice/crio-b8067af4ec64530c810f4d143f3be8aceb8e07525280f5a1bb1f98b829bea19d WatchSource:0}: Error finding container b8067af4ec64530c810f4d143f3be8aceb8e07525280f5a1bb1f98b829bea19d: Status 404 returned error can't find the container with id b8067af4ec64530c810f4d143f3be8aceb8e07525280f5a1bb1f98b829bea19d Nov 02 19:59:43.444692 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:43.444652 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" event={"ID":"c4bd8a7e-f968-44f5-a052-d37741cf404e","Type":"ContainerStarted","Data":"b8067af4ec64530c810f4d143f3be8aceb8e07525280f5a1bb1f98b829bea19d"} Nov 02 19:59:43.445908 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:43.445882 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586f56887b-8xt64" event={"ID":"433b226f-827b-40ef-ba68-2266e5ffca56","Type":"ContainerStarted","Data":"fc0eea14d14321abb3c5c51c5f03e462d31bc238029b0a1307089cb3746e3f43"} Nov 02 19:59:44.452498 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.452402 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" event={"ID":"1edb0b19-8c3f-486e-b0d8-05f471d57ff0","Type":"ContainerStarted","Data":"df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2"} Nov 02 19:59:44.452498 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.452456 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" podUID="1edb0b19-8c3f-486e-b0d8-05f471d57ff0" containerName="authorino" containerID="cri-o://df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2" gracePeriod=30 Nov 02 19:59:44.454078 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.454052 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586f56887b-8xt64" event={"ID":"433b226f-827b-40ef-ba68-2266e5ffca56","Type":"ContainerStarted","Data":"4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1"} Nov 02 19:59:44.454232 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.454185 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-586f56887b-8xt64" podUID="433b226f-827b-40ef-ba68-2266e5ffca56" containerName="authorino" containerID="cri-o://4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1" gracePeriod=30 Nov 02 19:59:44.559560 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.559496 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" podStartSLOduration=2.136638601 podStartE2EDuration="3.559475974s" totalImagesPullingTime="1.422837373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:44.558344459 +0000 UTC m=+427.168996550" watchObservedRunningTime="2026-11-02 19:59:44.559475974 +0000 UTC m=+427.170128066" Nov 02 19:59:44.666861 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.666807 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-586f56887b-8xt64" podStartSLOduration=1.3542241289999999 podStartE2EDuration="2.666788687s" totalImagesPullingTime="1.312564558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:44.66498358 +0000 UTC m=+427.275635672" watchObservedRunningTime="2026-11-02 19:59:44.666788687 +0000 UTC m=+427.277440785" Nov 02 19:59:44.729671 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.729639 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" Nov 02 19:59:44.737775 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.737751 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586f56887b-8xt64" Nov 02 19:59:44.772104 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.772077 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edb0b19-8c3f-486e-b0d8-05f471d57ff0-kube-api-access-cz8bz\" (UniqueName: \"kubernetes.io/projected/1edb0b19-8c3f-486e-b0d8-05f471d57ff0-kube-api-access-cz8bz\") pod \"1edb0b19-8c3f-486e-b0d8-05f471d57ff0\" (UID: \"1edb0b19-8c3f-486e-b0d8-05f471d57ff0\") " Nov 02 19:59:44.772312 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.772204 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433b226f-827b-40ef-ba68-2266e5ffca56-kube-api-access-z2mcp\" (UniqueName: \"kubernetes.io/projected/433b226f-827b-40ef-ba68-2266e5ffca56-kube-api-access-z2mcp\") pod \"433b226f-827b-40ef-ba68-2266e5ffca56\" (UID: \"433b226f-827b-40ef-ba68-2266e5ffca56\") " Nov 02 19:59:44.774148 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.774115 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edb0b19-8c3f-486e-b0d8-05f471d57ff0-kube-api-access-cz8bz" pod "1edb0b19-8c3f-486e-b0d8-05f471d57ff0" (UID: "1edb0b19-8c3f-486e-b0d8-05f471d57ff0"). InnerVolumeSpecName "kube-api-access-cz8bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:59:44.774250 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.774152 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433b226f-827b-40ef-ba68-2266e5ffca56-kube-api-access-z2mcp" pod "433b226f-827b-40ef-ba68-2266e5ffca56" (UID: "433b226f-827b-40ef-ba68-2266e5ffca56"). InnerVolumeSpecName "kube-api-access-z2mcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:59:44.873231 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.873192 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2mcp\" (UniqueName: \"kubernetes.io/projected/433b226f-827b-40ef-ba68-2266e5ffca56-kube-api-access-z2mcp\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:44.873231 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:44.873224 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz8bz\" (UniqueName: \"kubernetes.io/projected/1edb0b19-8c3f-486e-b0d8-05f471d57ff0-kube-api-access-cz8bz\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:45.474827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.474711 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" event={"ID":"c4bd8a7e-f968-44f5-a052-d37741cf404e","Type":"ContainerStarted","Data":"cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3"} Nov 02 19:59:45.474827 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.474748 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" podUID="c4bd8a7e-f968-44f5-a052-d37741cf404e" containerName="authorino" containerID="cri-o://cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3" gracePeriod=30 Nov 02 19:59:45.476884 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.476837 5399 generic.go:356] "Generic (PLEG): container finished" podID="1edb0b19-8c3f-486e-b0d8-05f471d57ff0" containerID="df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2" exitCode=0 Nov 02 19:59:45.477005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.476908 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" event={"ID":"1edb0b19-8c3f-486e-b0d8-05f471d57ff0","Type":"ContainerDied","Data":"df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2"} Nov 02 19:59:45.477005 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.476948 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" Nov 02 19:59:45.477134 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.477011 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lpwdx" event={"ID":"1edb0b19-8c3f-486e-b0d8-05f471d57ff0","Type":"ContainerDied","Data":"eec0994eeb7384549e96bf79de6a9e5e0b257ef47e8ebc0873548e8b183ee3c7"} Nov 02 19:59:45.477134 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.477037 5399 scope.go:122] "RemoveContainer" containerID="df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2" Nov 02 19:59:45.479765 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.479739 5399 generic.go:356] "Generic (PLEG): container finished" podID="433b226f-827b-40ef-ba68-2266e5ffca56" containerID="4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1" exitCode=0 Nov 02 19:59:45.479876 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.479772 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586f56887b-8xt64" event={"ID":"433b226f-827b-40ef-ba68-2266e5ffca56","Type":"ContainerDied","Data":"4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1"} Nov 02 19:59:45.479876 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.479800 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586f56887b-8xt64" Nov 02 19:59:45.479975 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.479808 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-586f56887b-8xt64" event={"ID":"433b226f-827b-40ef-ba68-2266e5ffca56","Type":"ContainerDied","Data":"fc0eea14d14321abb3c5c51c5f03e462d31bc238029b0a1307089cb3746e3f43"} Nov 02 19:59:45.486714 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.486691 5399 scope.go:122] "RemoveContainer" containerID="df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2" Nov 02 19:59:45.487033 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:59:45.487011 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2\": container with ID starting with df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2 not found: ID does not exist" containerID="df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2" Nov 02 19:59:45.487139 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.487044 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2"} err="failed to get container status \"df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2\": rpc error: code = NotFound desc = could not find container \"df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2\": container with ID starting with df14860ba2d86a9d54dc6ea6d2ed8e8bd27d090cfb84db2c57dc501f3800e1e2 not found: ID does not exist" Nov 02 19:59:45.487139 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.487067 5399 scope.go:122] "RemoveContainer" containerID="4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1" Nov 02 19:59:45.553549 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.553531 5399 scope.go:122] "RemoveContainer" containerID="4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1" Nov 02 19:59:45.553885 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:59:45.553864 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1\": container with ID starting with 4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1 not found: ID does not exist" containerID="4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1" Nov 02 19:59:45.553958 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.553900 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1"} err="failed to get container status \"4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1\": rpc error: code = NotFound desc = could not find container \"4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1\": container with ID starting with 4d5cd32e462c7198981812647792309aa0ef7549a6300449e388549699bfa8c1 not found: ID does not exist" Nov 02 19:59:45.580768 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.580718 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" podStartSLOduration=2.211945599 podStartE2EDuration="3.580704783s" totalImagesPullingTime="1.368759184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:45.580477204 +0000 UTC m=+428.191129296" watchObservedRunningTime="2026-11-02 19:59:45.580704783 +0000 UTC m=+428.191356874" Nov 02 19:59:45.684593 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.684555 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lpwdx"] Nov 02 19:59:45.689471 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.689443 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lpwdx"] Nov 02 19:59:45.722488 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.722462 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" Nov 02 19:59:45.781401 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.781321 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bd8a7e-f968-44f5-a052-d37741cf404e-kube-api-access-l8p2v\" (UniqueName: \"kubernetes.io/projected/c4bd8a7e-f968-44f5-a052-d37741cf404e-kube-api-access-l8p2v\") pod \"c4bd8a7e-f968-44f5-a052-d37741cf404e\" (UID: \"c4bd8a7e-f968-44f5-a052-d37741cf404e\") " Nov 02 19:59:45.781401 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.781356 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4bd8a7e-f968-44f5-a052-d37741cf404e-tls-cert\" (UniqueName: \"kubernetes.io/secret/c4bd8a7e-f968-44f5-a052-d37741cf404e-tls-cert\") pod \"c4bd8a7e-f968-44f5-a052-d37741cf404e\" (UID: \"c4bd8a7e-f968-44f5-a052-d37741cf404e\") " Nov 02 19:59:45.783703 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.783670 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bd8a7e-f968-44f5-a052-d37741cf404e-kube-api-access-l8p2v" pod "c4bd8a7e-f968-44f5-a052-d37741cf404e" (UID: "c4bd8a7e-f968-44f5-a052-d37741cf404e"). InnerVolumeSpecName "kube-api-access-l8p2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:59:45.791499 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.791473 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4bd8a7e-f968-44f5-a052-d37741cf404e-tls-cert" pod "c4bd8a7e-f968-44f5-a052-d37741cf404e" (UID: "c4bd8a7e-f968-44f5-a052-d37741cf404e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:59:45.882651 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.882615 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8p2v\" (UniqueName: \"kubernetes.io/projected/c4bd8a7e-f968-44f5-a052-d37741cf404e-kube-api-access-l8p2v\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:45.882651 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:45.882643 5399 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4bd8a7e-f968-44f5-a052-d37741cf404e-tls-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:59:46.486633 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:46.486544 5399 generic.go:356] "Generic (PLEG): container finished" podID="c4bd8a7e-f968-44f5-a052-d37741cf404e" containerID="cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3" exitCode=0 Nov 02 19:59:46.486633 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:46.486589 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" event={"ID":"c4bd8a7e-f968-44f5-a052-d37741cf404e","Type":"ContainerDied","Data":"cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3"} Nov 02 19:59:46.486633 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:46.486600 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" Nov 02 19:59:46.487229 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:46.486637 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f9ff88c54-g5rmj" event={"ID":"c4bd8a7e-f968-44f5-a052-d37741cf404e","Type":"ContainerDied","Data":"b8067af4ec64530c810f4d143f3be8aceb8e07525280f5a1bb1f98b829bea19d"} Nov 02 19:59:46.487229 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:46.486661 5399 scope.go:122] "RemoveContainer" containerID="cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3" Nov 02 19:59:46.496384 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:46.496365 5399 scope.go:122] "RemoveContainer" containerID="cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3" Nov 02 19:59:46.496643 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:59:46.496623 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3\": container with ID starting with cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3 not found: ID does not exist" containerID="cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3" Nov 02 19:59:46.496683 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:46.496654 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3"} err="failed to get container status \"cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3\": rpc error: code = NotFound desc = could not find container \"cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3\": container with ID starting with cc751cf8e330ff9a3df040271506b2fb680b7e7b2697d4a18c92978703c68aa3 not found: ID does not exist" Nov 02 19:59:46.637851 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:46.637815 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f9ff88c54-g5rmj"] Nov 02 19:59:46.641238 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:46.641212 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f9ff88c54-g5rmj"] Nov 02 19:59:47.505785 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:47.505748 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edb0b19-8c3f-486e-b0d8-05f471d57ff0" path="/var/lib/kubelet/pods/1edb0b19-8c3f-486e-b0d8-05f471d57ff0/volumes" Nov 02 19:59:47.506397 ip-10-0-130-111 kubenswrapper[5399]: I1102 19:59:47.506139 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4bd8a7e-f968-44f5-a052-d37741cf404e" path="/var/lib/kubelet/pods/c4bd8a7e-f968-44f5-a052-d37741cf404e/volumes" Nov 02 19:59:48.235028 ip-10-0-130-111 kubenswrapper[5399]: E1102 19:59:48.234992 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:00:00.154251 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.154217 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl"] Nov 02 20:00:00.154635 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.154598 5399 cpu_manager.go:414] "containerMap: removing container" podUID="1edb0b19-8c3f-486e-b0d8-05f471d57ff0" containerName="authorino" Nov 02 20:00:00.154635 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.154608 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edb0b19-8c3f-486e-b0d8-05f471d57ff0" containerName="authorino" Nov 02 20:00:00.154635 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.154619 5399 cpu_manager.go:414] "containerMap: removing container" podUID="433b226f-827b-40ef-ba68-2266e5ffca56" containerName="authorino" Nov 02 20:00:00.154635 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.154624 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433b226f-827b-40ef-ba68-2266e5ffca56" containerName="authorino" Nov 02 20:00:00.154758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.154638 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c4bd8a7e-f968-44f5-a052-d37741cf404e" containerName="authorino" Nov 02 20:00:00.154758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.154643 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4bd8a7e-f968-44f5-a052-d37741cf404e" containerName="authorino" Nov 02 20:00:00.154758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.154705 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4bd8a7e-f968-44f5-a052-d37741cf404e" containerName="authorino" Nov 02 20:00:00.154758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.154717 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edb0b19-8c3f-486e-b0d8-05f471d57ff0" containerName="authorino" Nov 02 20:00:00.154758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.154722 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="433b226f-827b-40ef-ba68-2266e5ffca56" containerName="authorino" Nov 02 20:00:00.157917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.157901 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" Nov 02 20:00:00.207184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.207144 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:00:00.216570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.216542 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33783c81-3f52-447b-bef7-748e52b91a64-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894160-t8lxl\" (UID: \"33783c81-3f52-447b-bef7-748e52b91a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" Nov 02 20:00:00.216698 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.216605 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6799q\" (UniqueName: \"kubernetes.io/projected/33783c81-3f52-447b-bef7-748e52b91a64-kube-api-access-6799q\") pod \"maas-api-key-cleanup-29894160-t8lxl\" (UID: \"33783c81-3f52-447b-bef7-748e52b91a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" Nov 02 20:00:00.253650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.253616 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:00:00.253811 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.253661 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:00:00.255314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.255292 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl"] Nov 02 20:00:00.317788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.317756 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6799q\" (UniqueName: \"kubernetes.io/projected/33783c81-3f52-447b-bef7-748e52b91a64-kube-api-access-6799q\") pod \"maas-api-key-cleanup-29894160-t8lxl\" (UID: \"33783c81-3f52-447b-bef7-748e52b91a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" Nov 02 20:00:00.317958 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.317843 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33783c81-3f52-447b-bef7-748e52b91a64-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894160-t8lxl\" (UID: \"33783c81-3f52-447b-bef7-748e52b91a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" Nov 02 20:00:00.318550 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.318532 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33783c81-3f52-447b-bef7-748e52b91a64-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894160-t8lxl\" (UID: \"33783c81-3f52-447b-bef7-748e52b91a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" Nov 02 20:00:00.371313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.371264 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6799q\" (UniqueName: \"kubernetes.io/projected/33783c81-3f52-447b-bef7-748e52b91a64-kube-api-access-6799q\") pod \"maas-api-key-cleanup-29894160-t8lxl\" (UID: \"33783c81-3f52-447b-bef7-748e52b91a64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" Nov 02 20:00:00.469737 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.469658 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" Nov 02 20:00:00.829126 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:00.829053 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl"] Nov 02 20:00:00.831412 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:00:00.831371 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33783c81_3f52_447b_bef7_748e52b91a64.slice/crio-9605e57fcf3fd7d2098c1bef36cca1d68aaa068416c8021af2ca8a08ed094b17 WatchSource:0}: Error finding container 9605e57fcf3fd7d2098c1bef36cca1d68aaa068416c8021af2ca8a08ed094b17: Status 404 returned error can't find the container with id 9605e57fcf3fd7d2098c1bef36cca1d68aaa068416c8021af2ca8a08ed094b17 Nov 02 20:00:01.564161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:01.564120 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" event={"ID":"33783c81-3f52-447b-bef7-748e52b91a64","Type":"ContainerStarted","Data":"9605e57fcf3fd7d2098c1bef36cca1d68aaa068416c8021af2ca8a08ed094b17"} Nov 02 20:00:02.393846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:02.393812 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 20:00:02.397007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:02.396982 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-lmtgk" Nov 02 20:00:07.596725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:07.596687 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" event={"ID":"33783c81-3f52-447b-bef7-748e52b91a64","Type":"ContainerStarted","Data":"cc23f12df2f14b6524ec6a08fd85d551a1da007e8284b7d2b1849e89468f1656"} Nov 02 20:00:07.702785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:07.702730 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" podStartSLOduration=1.292677318 podStartE2EDuration="7.702715654s" totalImagesPullingTime="6.410038336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:07.700438207 +0000 UTC m=+450.311090298" watchObservedRunningTime="2026-11-02 20:00:07.702715654 +0000 UTC m=+450.313367762" Nov 02 20:00:08.603349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:08.603313 5399 generic.go:356] "Generic (PLEG): container finished" podID="33783c81-3f52-447b-bef7-748e52b91a64" containerID="cc23f12df2f14b6524ec6a08fd85d551a1da007e8284b7d2b1849e89468f1656" exitCode=60 Nov 02 20:00:08.603790 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:08.603405 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" event={"ID":"33783c81-3f52-447b-bef7-748e52b91a64","Type":"ContainerDied","Data":"cc23f12df2f14b6524ec6a08fd85d551a1da007e8284b7d2b1849e89468f1656"} Nov 02 20:00:08.603790 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:08.603600 5399 scope.go:122] "RemoveContainer" containerID="cc23f12df2f14b6524ec6a08fd85d551a1da007e8284b7d2b1849e89468f1656" Nov 02 20:00:09.610298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:09.610189 5399 generic.go:356] "Generic (PLEG): container finished" podID="33783c81-3f52-447b-bef7-748e52b91a64" containerID="b95eb61ffa502dbd1f627b3da6ab839d93c150d7cbeb0d37cfd2120c3d4f6408" exitCode=60 Nov 02 20:00:09.610298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:09.610246 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" event={"ID":"33783c81-3f52-447b-bef7-748e52b91a64","Type":"ContainerDied","Data":"b95eb61ffa502dbd1f627b3da6ab839d93c150d7cbeb0d37cfd2120c3d4f6408"} Nov 02 20:00:09.610756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:09.610325 5399 scope.go:122] "RemoveContainer" containerID="cc23f12df2f14b6524ec6a08fd85d551a1da007e8284b7d2b1849e89468f1656" Nov 02 20:00:09.610756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:09.610518 5399 scope.go:122] "RemoveContainer" containerID="b95eb61ffa502dbd1f627b3da6ab839d93c150d7cbeb0d37cfd2120c3d4f6408" Nov 02 20:00:09.610827 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:00:09.610767 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894160-t8lxl_odh-ai-gateway-infra(33783c81-3f52-447b-bef7-748e52b91a64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" podUID="33783c81-3f52-447b-bef7-748e52b91a64" Nov 02 20:00:10.617195 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:10.617160 5399 scope.go:122] "RemoveContainer" containerID="b95eb61ffa502dbd1f627b3da6ab839d93c150d7cbeb0d37cfd2120c3d4f6408" Nov 02 20:00:10.617666 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:00:10.617432 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894160-t8lxl_odh-ai-gateway-infra(33783c81-3f52-447b-bef7-748e52b91a64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" podUID="33783c81-3f52-447b-bef7-748e52b91a64" Nov 02 20:00:15.549348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:15.549297 5399 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","besteffort","pod433b226f-827b-40ef-ba68-2266e5ffca56"] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod433b226f-827b-40ef-ba68-2266e5ffca56] : Timed out while waiting for systemd to remove kubepods-besteffort-pod433b226f_827b_40ef_ba68_2266e5ffca56.slice" Nov 02 20:00:15.549751 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:00:15.549350 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods besteffort pod433b226f-827b-40ef-ba68-2266e5ffca56] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod433b226f-827b-40ef-ba68-2266e5ffca56] : Timed out while waiting for systemd to remove kubepods-besteffort-pod433b226f_827b_40ef_ba68_2266e5ffca56.slice" pod="kuadrant-system/authorino-586f56887b-8xt64" podUID="433b226f-827b-40ef-ba68-2266e5ffca56" Nov 02 20:00:15.640787 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:15.640755 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-586f56887b-8xt64" Nov 02 20:00:15.792112 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:15.792075 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-586f56887b-8xt64"] Nov 02 20:00:15.795116 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:15.795093 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-586f56887b-8xt64"] Nov 02 20:00:17.507457 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:17.507379 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="433b226f-827b-40ef-ba68-2266e5ffca56" path="/var/lib/kubelet/pods/433b226f-827b-40ef-ba68-2266e5ffca56/volumes" Nov 02 20:00:18.235122 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:00:18.235077 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:00:22.501199 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:22.501167 5399 scope.go:122] "RemoveContainer" containerID="b95eb61ffa502dbd1f627b3da6ab839d93c150d7cbeb0d37cfd2120c3d4f6408" Nov 02 20:00:23.680718 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:23.680684 5399 generic.go:356] "Generic (PLEG): container finished" podID="33783c81-3f52-447b-bef7-748e52b91a64" containerID="cc32f7b6aa28b9ceb193e884139e313b7a66c041257b0dab7733c453db4c6d45" exitCode=60 Nov 02 20:00:23.681142 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:23.680752 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" event={"ID":"33783c81-3f52-447b-bef7-748e52b91a64","Type":"ContainerDied","Data":"cc32f7b6aa28b9ceb193e884139e313b7a66c041257b0dab7733c453db4c6d45"} Nov 02 20:00:23.681142 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:23.680810 5399 scope.go:122] "RemoveContainer" containerID="b95eb61ffa502dbd1f627b3da6ab839d93c150d7cbeb0d37cfd2120c3d4f6408" Nov 02 20:00:23.681142 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:23.681122 5399 scope.go:122] "RemoveContainer" containerID="cc32f7b6aa28b9ceb193e884139e313b7a66c041257b0dab7733c453db4c6d45" Nov 02 20:00:23.681371 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:00:23.681349 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894160-t8lxl_odh-ai-gateway-infra(33783c81-3f52-447b-bef7-748e52b91a64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" podUID="33783c81-3f52-447b-bef7-748e52b91a64" Nov 02 20:00:24.796791 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:24.796756 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl"] Nov 02 20:00:24.923904 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:24.923878 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" Nov 02 20:00:25.056515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.043546 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33783c81-3f52-447b-bef7-748e52b91a64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33783c81-3f52-447b-bef7-748e52b91a64-maas-api-service-ca\") pod \"33783c81-3f52-447b-bef7-748e52b91a64\" (UID: \"33783c81-3f52-447b-bef7-748e52b91a64\") " Nov 02 20:00:25.056515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.043598 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33783c81-3f52-447b-bef7-748e52b91a64-kube-api-access-6799q\" (UniqueName: \"kubernetes.io/projected/33783c81-3f52-447b-bef7-748e52b91a64-kube-api-access-6799q\") pod \"33783c81-3f52-447b-bef7-748e52b91a64\" (UID: \"33783c81-3f52-447b-bef7-748e52b91a64\") " Nov 02 20:00:25.056515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.054774 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33783c81-3f52-447b-bef7-748e52b91a64-maas-api-service-ca" pod "33783c81-3f52-447b-bef7-748e52b91a64" (UID: "33783c81-3f52-447b-bef7-748e52b91a64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:25.056515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.055163 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33783c81-3f52-447b-bef7-748e52b91a64-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:25.057581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.057550 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33783c81-3f52-447b-bef7-748e52b91a64-kube-api-access-6799q" pod "33783c81-3f52-447b-bef7-748e52b91a64" (UID: "33783c81-3f52-447b-bef7-748e52b91a64"). InnerVolumeSpecName "kube-api-access-6799q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:25.156048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.156009 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6799q\" (UniqueName: \"kubernetes.io/projected/33783c81-3f52-447b-bef7-748e52b91a64-kube-api-access-6799q\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:25.695043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.695007 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" event={"ID":"33783c81-3f52-447b-bef7-748e52b91a64","Type":"ContainerDied","Data":"9605e57fcf3fd7d2098c1bef36cca1d68aaa068416c8021af2ca8a08ed094b17"} Nov 02 20:00:25.695043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.695024 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl" Nov 02 20:00:25.695297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.695077 5399 scope.go:122] "RemoveContainer" containerID="cc32f7b6aa28b9ceb193e884139e313b7a66c041257b0dab7733c453db4c6d45" Nov 02 20:00:25.853144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.853099 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl"] Nov 02 20:00:25.854869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:25.854846 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-t8lxl"] Nov 02 20:00:27.505524 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:00:27.505486 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33783c81-3f52-447b-bef7-748e52b91a64" path="/var/lib/kubelet/pods/33783c81-3f52-447b-bef7-748e52b91a64/volumes" Nov 02 20:00:48.234560 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:00:48.234525 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:01:18.235252 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:01:18.235215 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:01:25.396127 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.396098 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp"] Nov 02 20:01:25.396605 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.396316 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" podUID="0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714" containerName="manager" containerID="cri-o://a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90" gracePeriod=30 Nov 02 20:01:25.637105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.637048 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 20:01:25.766841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.766806 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-kube-api-access-gt6tl\" (UniqueName: \"kubernetes.io/projected/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-kube-api-access-gt6tl\") pod \"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714\" (UID: \"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714\") " Nov 02 20:01:25.767026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.766926 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-cert\" (UniqueName: \"kubernetes.io/secret/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-cert\") pod \"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714\" (UID: \"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714\") " Nov 02 20:01:25.768917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.768887 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-cert" pod "0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714" (UID: "0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:01:25.769085 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.769068 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-kube-api-access-gt6tl" pod "0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714" (UID: "0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714"). InnerVolumeSpecName "kube-api-access-gt6tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:01:25.868292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.868240 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6tl\" (UniqueName: \"kubernetes.io/projected/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-kube-api-access-gt6tl\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:25.868292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.868270 5399 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:01:25.976913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.976828 5399 generic.go:356] "Generic (PLEG): container finished" podID="0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714" containerID="a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90" exitCode=0 Nov 02 20:01:25.976913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.976890 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" Nov 02 20:01:25.977124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.976915 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" event={"ID":"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714","Type":"ContainerDied","Data":"a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90"} Nov 02 20:01:25.977124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.976995 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp" event={"ID":"0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714","Type":"ContainerDied","Data":"1f9fe0dd0e2692c0534ca761c13bf803aa58b1b573e0a808f516562f03b35021"} Nov 02 20:01:25.977124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.977025 5399 scope.go:122] "RemoveContainer" containerID="a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90" Nov 02 20:01:25.978672 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.978643 5399 generic.go:356] "Generic (PLEG): container finished" podID="ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" containerID="e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c" exitCode=1 Nov 02 20:01:25.978810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.978684 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" event={"ID":"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036","Type":"ContainerDied","Data":"e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c"} Nov 02 20:01:25.979032 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.979016 5399 scope.go:122] "RemoveContainer" containerID="e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c" Nov 02 20:01:25.986901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.986881 5399 scope.go:122] "RemoveContainer" containerID="a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90" Nov 02 20:01:25.987191 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:01:25.987165 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90\": container with ID starting with a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90 not found: ID does not exist" containerID="a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90" Nov 02 20:01:25.987264 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:25.987201 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90"} err="failed to get container status \"a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90\": rpc error: code = NotFound desc = could not find container \"a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90\": container with ID starting with a87543af436a65d503ed439983c1d4a477138a3ab1dfdc80c6fbf15206d0fd90 not found: ID does not exist" Nov 02 20:01:26.237762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:26.237669 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp"] Nov 02 20:01:26.239769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:26.239744 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2wbsp"] Nov 02 20:01:26.986326 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:26.986268 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" event={"ID":"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036","Type":"ContainerStarted","Data":"55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39"} Nov 02 20:01:26.986784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:26.986557 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 20:01:27.505778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:27.505741 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714" path="/var/lib/kubelet/pods/0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714/volumes" Nov 02 20:01:37.993265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:37.993182 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 20:01:48.235747 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:01:48.235658 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:01:50.649673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.649638 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr"] Nov 02 20:01:50.650046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.649967 5399 cpu_manager.go:414] "containerMap: removing container" podUID="33783c81-3f52-447b-bef7-748e52b91a64" containerName="cleanup" Nov 02 20:01:50.650046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.649978 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33783c81-3f52-447b-bef7-748e52b91a64" containerName="cleanup" Nov 02 20:01:50.650046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.649990 5399 cpu_manager.go:414] "containerMap: removing container" podUID="33783c81-3f52-447b-bef7-748e52b91a64" containerName="cleanup" Nov 02 20:01:50.650046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.649995 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33783c81-3f52-447b-bef7-748e52b91a64" containerName="cleanup" Nov 02 20:01:50.650046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.650014 5399 cpu_manager.go:414] "containerMap: removing container" podUID="0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714" containerName="manager" Nov 02 20:01:50.650046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.650019 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714" containerName="manager" Nov 02 20:01:50.650241 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.650080 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="33783c81-3f52-447b-bef7-748e52b91a64" containerName="cleanup" Nov 02 20:01:50.650241 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.650088 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="33783c81-3f52-447b-bef7-748e52b91a64" containerName="cleanup" Nov 02 20:01:50.650241 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.650097 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4ecfc2-2e24-4bc2-bf0a-c4056cbac714" containerName="manager" Nov 02 20:01:50.653057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.653043 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.735765 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.735731 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 20:01:50.750651 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.750625 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 20:01:50.750740 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.750724 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d2qwk\"" Nov 02 20:01:50.750786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.750732 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 20:01:50.752183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.752165 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr"] Nov 02 20:01:50.781672 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.781650 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.781763 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.781677 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/104f859d-9978-4548-91da-a1d3e1438dca-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.781763 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.781709 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.781763 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.781727 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnhl\" (UniqueName: \"kubernetes.io/projected/104f859d-9978-4548-91da-a1d3e1438dca-kube-api-access-8cnhl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.781868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.781831 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.781912 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.781863 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.882802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.882767 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.882802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.882801 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/104f859d-9978-4548-91da-a1d3e1438dca-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.883040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.882833 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.883040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.882850 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnhl\" (UniqueName: \"kubernetes.io/projected/104f859d-9978-4548-91da-a1d3e1438dca-kube-api-access-8cnhl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.883040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.882873 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.883040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.882930 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.883267 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.883247 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.883342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.883269 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.883342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.883320 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.885157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.885135 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/104f859d-9978-4548-91da-a1d3e1438dca-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.885418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.885402 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/104f859d-9978-4548-91da-a1d3e1438dca-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.935845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.935779 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnhl\" (UniqueName: \"kubernetes.io/projected/104f859d-9978-4548-91da-a1d3e1438dca-kube-api-access-8cnhl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr\" (UID: \"104f859d-9978-4548-91da-a1d3e1438dca\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:50.964929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:50.964895 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:51.325629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:51.325582 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr"] Nov 02 20:01:51.328320 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:01:51.328292 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104f859d_9978_4548_91da_a1d3e1438dca.slice/crio-7e372163ab4c0426ff00e0ee8ef42513680a4c47aaf7195ef0eb0af5d86020d5 WatchSource:0}: Error finding container 7e372163ab4c0426ff00e0ee8ef42513680a4c47aaf7195ef0eb0af5d86020d5: Status 404 returned error can't find the container with id 7e372163ab4c0426ff00e0ee8ef42513680a4c47aaf7195ef0eb0af5d86020d5 Nov 02 20:01:52.107874 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:52.107831 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" event={"ID":"104f859d-9978-4548-91da-a1d3e1438dca","Type":"ContainerStarted","Data":"7e372163ab4c0426ff00e0ee8ef42513680a4c47aaf7195ef0eb0af5d86020d5"} Nov 02 20:01:53.113614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:53.113581 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" event={"ID":"104f859d-9978-4548-91da-a1d3e1438dca","Type":"ContainerStarted","Data":"8f641a8e2c80c243710f8f5bcad27a0091b91a1fcdd6cc83a67e6e1093b4a029"} Nov 02 20:01:53.220570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:53.220466 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" podStartSLOduration=1.6520610279999999 podStartE2EDuration="3.220451042s" totalImagesPullingTime="1.568390014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:53.218972212 +0000 UTC m=+555.829624303" watchObservedRunningTime="2026-11-02 20:01:53.220451042 +0000 UTC m=+555.831103132" Nov 02 20:01:53.668812 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:53.668776 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:01:53.965105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:53.965023 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:01:57.983897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:57.983856 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg"] Nov 02 20:01:57.984941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:57.984914 5399 cpu_manager.go:414] "containerMap: removing container" podUID="33783c81-3f52-447b-bef7-748e52b91a64" containerName="cleanup" Nov 02 20:01:57.984941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:57.984940 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33783c81-3f52-447b-bef7-748e52b91a64" containerName="cleanup" Nov 02 20:01:57.985115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:57.985106 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="33783c81-3f52-447b-bef7-748e52b91a64" containerName="cleanup" Nov 02 20:01:57.989468 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:57.989446 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.048586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.048555 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.048773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.048591 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f7e6ae-694c-43dc-b392-a6414481c8cf-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.048773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.048698 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcv9d\" (UniqueName: \"kubernetes.io/projected/d2f7e6ae-694c-43dc-b392-a6414481c8cf-kube-api-access-wcv9d\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.048907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.048862 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.048907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.048895 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.049057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.048986 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.087041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.087009 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 20:01:58.088627 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.088607 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg"] Nov 02 20:01:58.150444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.150409 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.150623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.150459 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.150623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.150475 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f7e6ae-694c-43dc-b392-a6414481c8cf-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.150623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.150594 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcv9d\" (UniqueName: \"kubernetes.io/projected/d2f7e6ae-694c-43dc-b392-a6414481c8cf-kube-api-access-wcv9d\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.150742 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.150691 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.150742 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.150724 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.150849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.150816 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.151031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.151007 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.151154 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.151131 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.152691 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.152669 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f7e6ae-694c-43dc-b392-a6414481c8cf-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.152984 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.152969 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f7e6ae-694c-43dc-b392-a6414481c8cf-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.203435 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.203406 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcv9d\" (UniqueName: \"kubernetes.io/projected/d2f7e6ae-694c-43dc-b392-a6414481c8cf-kube-api-access-wcv9d\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-d5kgg\" (UID: \"d2f7e6ae-694c-43dc-b392-a6414481c8cf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.302592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.302507 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:01:58.665780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:58.665749 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg"] Nov 02 20:01:58.668999 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:01:58.668971 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f7e6ae_694c_43dc_b392_a6414481c8cf.slice/crio-f296e9bcb57efe65060bb4d953d5e6a090304a8fadf8014d3555e01aaf2c26ad WatchSource:0}: Error finding container f296e9bcb57efe65060bb4d953d5e6a090304a8fadf8014d3555e01aaf2c26ad: Status 404 returned error can't find the container with id f296e9bcb57efe65060bb4d953d5e6a090304a8fadf8014d3555e01aaf2c26ad Nov 02 20:01:59.142964 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:01:59.142927 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" event={"ID":"d2f7e6ae-694c-43dc-b392-a6414481c8cf","Type":"ContainerStarted","Data":"f296e9bcb57efe65060bb4d953d5e6a090304a8fadf8014d3555e01aaf2c26ad"} Nov 02 20:02:00.148981 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:00.148948 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" event={"ID":"d2f7e6ae-694c-43dc-b392-a6414481c8cf","Type":"ContainerStarted","Data":"b934f8df80afff43b7e4d8ad241dbaa90c10aff86306f20b3d2dffa830af72d0"} Nov 02 20:02:00.255997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:00.255945 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" podStartSLOduration=2.653891401 podStartE2EDuration="3.255930992s" totalImagesPullingTime="602.039591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:00.255046802 +0000 UTC m=+562.865698890" watchObservedRunningTime="2026-11-02 20:02:00.255930992 +0000 UTC m=+562.866583083" Nov 02 20:02:00.303621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:00.303577 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:02:00.965746 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:00.965707 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:02:00.978167 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:00.978143 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:02:01.165455 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:01.165429 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" Nov 02 20:02:01.232330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:01.232235 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:02:04.446071 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.446028 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr"] Nov 02 20:02:04.449704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.449688 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.527592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.527554 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr"] Nov 02 20:02:04.547150 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.547122 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 20:02:04.607161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.607127 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a980d-dfca-4c6e-964f-38a7496267fe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.607372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.607239 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.607372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.607305 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.607372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.607333 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cblf\" (UniqueName: \"kubernetes.io/projected/930a980d-dfca-4c6e-964f-38a7496267fe-kube-api-access-6cblf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.607372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.607353 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.607549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.607478 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.708929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.708843 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.708929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.708883 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a980d-dfca-4c6e-964f-38a7496267fe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.708929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.708927 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.709203 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.708953 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.709203 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.708976 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cblf\" (UniqueName: \"kubernetes.io/projected/930a980d-dfca-4c6e-964f-38a7496267fe-kube-api-access-6cblf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.709203 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.709007 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.709374 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.709240 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.709420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.709401 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.709460 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.709433 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.711235 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.711210 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a980d-dfca-4c6e-964f-38a7496267fe-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.711743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.711725 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a980d-dfca-4c6e-964f-38a7496267fe-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:04.764723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:04.764695 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cblf\" (UniqueName: \"kubernetes.io/projected/930a980d-dfca-4c6e-964f-38a7496267fe-kube-api-access-6cblf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mkqnr\" (UID: \"930a980d-dfca-4c6e-964f-38a7496267fe\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:05.062247 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:05.062217 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:05.425757 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:05.425733 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr"] Nov 02 20:02:05.428426 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:02:05.428398 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930a980d_dfca_4c6e_964f_38a7496267fe.slice/crio-dc9c7299700cf8572132ee4dfa19ea457c667c77713fecc3040ad1ce9bf07c4e WatchSource:0}: Error finding container dc9c7299700cf8572132ee4dfa19ea457c667c77713fecc3040ad1ce9bf07c4e: Status 404 returned error can't find the container with id dc9c7299700cf8572132ee4dfa19ea457c667c77713fecc3040ad1ce9bf07c4e Nov 02 20:02:06.179125 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:06.179092 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" event={"ID":"930a980d-dfca-4c6e-964f-38a7496267fe","Type":"ContainerStarted","Data":"bba097229476b8b5f01fc40ed68eef0c17809152ad1572b7c8ee3b091822bc8b"} Nov 02 20:02:06.179593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:06.179152 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" event={"ID":"930a980d-dfca-4c6e-964f-38a7496267fe","Type":"ContainerStarted","Data":"dc9c7299700cf8572132ee4dfa19ea457c667c77713fecc3040ad1ce9bf07c4e"} Nov 02 20:02:06.286339 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:06.286268 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" podStartSLOduration=1.639761789 podStartE2EDuration="2.286255648s" totalImagesPullingTime="646.493859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:06.284945996 +0000 UTC m=+568.895598087" watchObservedRunningTime="2026-11-02 20:02:06.286255648 +0000 UTC m=+568.896907739" Nov 02 20:02:07.063119 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:07.063073 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:08.303019 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:08.302985 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:02:08.315674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:08.315648 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:02:09.204519 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:09.204492 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" Nov 02 20:02:12.092026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.091988 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dxf74"] Nov 02 20:02:12.098493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.098470 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.149714 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.149681 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dxf74"] Nov 02 20:02:12.175270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.175240 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.175440 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.175370 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-home\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.175440 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.175406 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.175511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.175491 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.175544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.175516 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.175544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.175535 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hczwq\" (UniqueName: \"kubernetes.io/projected/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-kube-api-access-hczwq\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.193518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.193490 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 20:02:12.276846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.276812 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-home\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.277031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.276854 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.277031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.276899 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.277031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.276918 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.277031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.276940 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hczwq\" (UniqueName: \"kubernetes.io/projected/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-kube-api-access-hczwq\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.277031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.276961 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.277392 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.277368 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.277510 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.277467 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.277565 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.277469 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-home\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.279187 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.279164 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.279515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.279497 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.329863 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.329834 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hczwq\" (UniqueName: \"kubernetes.io/projected/7d6e94b5-d4a7-48d4-ab4b-27b457ed367c-kube-api-access-hczwq\") pod \"echo-simulator-kserve-64b6f8f96c-dxf74\" (UID: \"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.411599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.411505 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:12.778938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:12.778912 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dxf74"] Nov 02 20:02:12.784073 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:02:12.784026 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6e94b5_d4a7_48d4_ab4b_27b457ed367c.slice/crio-90f6222bfabc6277273d2787dda47a3d09979162293208b20b49445623fbfbb9 WatchSource:0}: Error finding container 90f6222bfabc6277273d2787dda47a3d09979162293208b20b49445623fbfbb9: Status 404 returned error can't find the container with id 90f6222bfabc6277273d2787dda47a3d09979162293208b20b49445623fbfbb9 Nov 02 20:02:13.215836 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:13.215801 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" event={"ID":"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c","Type":"ContainerStarted","Data":"90f6222bfabc6277273d2787dda47a3d09979162293208b20b49445623fbfbb9"} Nov 02 20:02:15.062969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:15.062933 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:15.076981 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:15.076955 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:15.238879 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:15.238852 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" Nov 02 20:02:15.932234 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:15.932201 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:02:15.978526 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:15.978492 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj"] Nov 02 20:02:15.982062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:15.982037 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.034665 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.034622 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 20:02:16.035463 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.035440 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj"] Nov 02 20:02:16.109506 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.109474 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.109992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.109529 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.109992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.109619 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87db511f-cd55-456d-9b98-af4df0327cfc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.109992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.109750 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.109992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.109772 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.109992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.109865 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vwc7\" (UniqueName: \"kubernetes.io/projected/87db511f-cd55-456d-9b98-af4df0327cfc-kube-api-access-5vwc7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.210500 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.210413 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87db511f-cd55-456d-9b98-af4df0327cfc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.210500 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.210470 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.210500 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.210485 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.210760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.210513 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vwc7\" (UniqueName: \"kubernetes.io/projected/87db511f-cd55-456d-9b98-af4df0327cfc-kube-api-access-5vwc7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.210760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.210546 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.210760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.210608 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.210948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.210926 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.210948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.210940 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.211062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.211023 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.212825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.212799 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87db511f-cd55-456d-9b98-af4df0327cfc-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.212924 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.212900 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87db511f-cd55-456d-9b98-af4df0327cfc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.232817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.232782 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" event={"ID":"7d6e94b5-d4a7-48d4-ab4b-27b457ed367c","Type":"ContainerStarted","Data":"ff6ee3a02f2dc16187c3b3b800c5634366ae9cd3b2ba10b10d9c43d751d53c6a"} Nov 02 20:02:16.263651 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.263623 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vwc7\" (UniqueName: \"kubernetes.io/projected/87db511f-cd55-456d-9b98-af4df0327cfc-kube-api-access-5vwc7\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj\" (UID: \"87db511f-cd55-456d-9b98-af4df0327cfc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.294995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.294968 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:16.339662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.339595 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" podStartSLOduration=1.7007670240000001 podStartE2EDuration="4.339576808s" totalImagesPullingTime="2.638809784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:16.338782886 +0000 UTC m=+578.949434980" watchObservedRunningTime="2026-11-02 20:02:16.339576808 +0000 UTC m=+578.950228899" Nov 02 20:02:16.412260 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.412222 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:16.657916 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:16.657891 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj"] Nov 02 20:02:16.659891 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:02:16.659855 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87db511f_cd55_456d_9b98_af4df0327cfc.slice/crio-07f16eaf1308c0b0b2610e3d9bd171468697ebe7ef20c55ffa87c5ad745d8b60 WatchSource:0}: Error finding container 07f16eaf1308c0b0b2610e3d9bd171468697ebe7ef20c55ffa87c5ad745d8b60: Status 404 returned error can't find the container with id 07f16eaf1308c0b0b2610e3d9bd171468697ebe7ef20c55ffa87c5ad745d8b60 Nov 02 20:02:17.239944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:17.239908 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" event={"ID":"87db511f-cd55-456d-9b98-af4df0327cfc","Type":"ContainerStarted","Data":"07f16eaf1308c0b0b2610e3d9bd171468697ebe7ef20c55ffa87c5ad745d8b60"} Nov 02 20:02:18.236495 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:02:18.236456 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:02:18.246424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:18.246390 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" event={"ID":"87db511f-cd55-456d-9b98-af4df0327cfc","Type":"ContainerStarted","Data":"f1cb9e9d6bbb0c80dc390122c730b0210eefb13d8bf2a78903db791878591386"} Nov 02 20:02:18.295085 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:18.295052 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:18.354168 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:18.354104 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" podStartSLOduration=2.718972166 podStartE2EDuration="3.354090766s" totalImagesPullingTime="635.1186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:18.352206481 +0000 UTC m=+580.962858609" watchObservedRunningTime="2026-11-02 20:02:18.354090766 +0000 UTC m=+580.964742857" Nov 02 20:02:22.412084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:22.412049 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:22.424654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:22.424630 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:23.282704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:23.282677 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" Nov 02 20:02:26.295768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:26.295735 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:26.308520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:26.308494 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:27.302097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:27.302071 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj" Nov 02 20:02:28.661336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.661296 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq"] Nov 02 20:02:28.664484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.664469 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.713340 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.713317 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 20:02:28.726405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.726375 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.726525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.726409 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.726525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.726439 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.726607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.726554 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.726607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.726582 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.726716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.726608 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.726716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.726628 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.726815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.726769 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953c331-dda5-4f9e-b084-27e3b2caa37c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.726855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.726834 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhgwq\" (UniqueName: \"kubernetes.io/projected/9953c331-dda5-4f9e-b084-27e3b2caa37c-kube-api-access-fhgwq\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.759629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.759600 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fdd8j\"" Nov 02 20:02:28.761750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.761728 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq"] Nov 02 20:02:28.827655 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.827620 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.827836 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.827671 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.827836 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.827714 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.827836 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.827734 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.827836 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.827771 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.827836 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.827802 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.828180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.827867 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953c331-dda5-4f9e-b084-27e3b2caa37c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.828180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.827893 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhgwq\" (UniqueName: \"kubernetes.io/projected/9953c331-dda5-4f9e-b084-27e3b2caa37c-kube-api-access-fhgwq\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.828180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.827947 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.828180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.828125 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.828180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.828143 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.828605 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.828580 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953c331-dda5-4f9e-b084-27e3b2caa37c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.828736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.828714 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.828789 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.828748 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.830369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.830343 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.830494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.830472 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.880535 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.880504 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953c331-dda5-4f9e-b084-27e3b2caa37c-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.880677 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.880587 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhgwq\" (UniqueName: \"kubernetes.io/projected/9953c331-dda5-4f9e-b084-27e3b2caa37c-kube-api-access-fhgwq\") pod \"maas-default-gateway-openshift-default-696f47c6c-g7smq\" (UID: \"9953c331-dda5-4f9e-b084-27e3b2caa37c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:28.979087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:28.978866 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:29.390157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:29.390130 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq"] Nov 02 20:02:29.391339 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:02:29.391307 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9953c331_dda5_4f9e_b084_27e3b2caa37c.slice/crio-96174bdacc86bb9b16dc052b064a19cd841fcdef34e9f036a24a1016f3d5c9de WatchSource:0}: Error finding container 96174bdacc86bb9b16dc052b064a19cd841fcdef34e9f036a24a1016f3d5c9de: Status 404 returned error can't find the container with id 96174bdacc86bb9b16dc052b064a19cd841fcdef34e9f036a24a1016f3d5c9de Nov 02 20:02:29.439398 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:29.439372 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:02:30.307305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:30.307261 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" event={"ID":"9953c331-dda5-4f9e-b084-27e3b2caa37c","Type":"ContainerStarted","Data":"96174bdacc86bb9b16dc052b064a19cd841fcdef34e9f036a24a1016f3d5c9de"} Nov 02 20:02:33.452631 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:33.452598 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:02:33.452896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:33.452666 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:02:33.452896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:33.452695 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:02:34.356689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:34.356649 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" event={"ID":"9953c331-dda5-4f9e-b084-27e3b2caa37c","Type":"ContainerStarted","Data":"21a88d3111e5087babf4ae481b1b9f847dc936c805abf5374add15010830f1de"} Nov 02 20:02:34.512592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:34.512538 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" podStartSLOduration=2.454106518 podStartE2EDuration="6.512522193s" totalImagesPullingTime="4.058415675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:34.510971463 +0000 UTC m=+597.121623555" watchObservedRunningTime="2026-11-02 20:02:34.512522193 +0000 UTC m=+597.123174283" Nov 02 20:02:34.979168 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:34.979129 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:34.980716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:34.980686 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-g7smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:02:34.980841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:34.980749 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" podUID="9953c331-dda5-4f9e-b084-27e3b2caa37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:02:35.979111 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:35.979076 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-g7smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:02:35.979648 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:35.979136 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" podUID="9953c331-dda5-4f9e-b084-27e3b2caa37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:02:36.979959 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:36.979915 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-g7smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:02:36.980439 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:36.979979 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" podUID="9953c331-dda5-4f9e-b084-27e3b2caa37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:02:37.979188 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:37.979144 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-g7smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:02:37.979431 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:37.979223 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" podUID="9953c331-dda5-4f9e-b084-27e3b2caa37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:02:38.979583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:38.979544 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-g7smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:02:38.980086 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:38.979608 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" podUID="9953c331-dda5-4f9e-b084-27e3b2caa37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:02:39.980221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:39.980182 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-g7smq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:02:39.980630 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:39.980260 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" podUID="9953c331-dda5-4f9e-b084-27e3b2caa37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:02:41.652392 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:41.652360 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:41.652888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:41.652792 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:41.653601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:41.653579 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" Nov 02 20:02:43.788261 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.788225 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj"] Nov 02 20:02:43.792967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.792946 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:43.880798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.880761 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj"] Nov 02 20:02:43.901749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.901715 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:43.901925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.901779 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:43.901925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.901843 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:43.902006 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.901964 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:43.902006 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.901996 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjqtg\" (UniqueName: \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-kube-api-access-mjqtg\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:43.902078 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.902068 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:43.902114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.902088 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:43.902173 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.902117 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:43.902258 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:43.902239 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.002726 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.002691 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.002726 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.002723 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.002988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.002750 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.002988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.002796 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.002988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.002838 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.002988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.002880 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.002988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.002932 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.002988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.002962 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.002988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.002987 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjqtg\" (UniqueName: \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-kube-api-access-mjqtg\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.003371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.003242 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.003371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.003269 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.003371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.003271 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.003614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.003594 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.003837 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.003818 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.005259 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.005242 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.005474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.005457 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.055967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.055931 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.056112 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.056082 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjqtg\" (UniqueName: \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-kube-api-access-mjqtg\") pod \"maas-default-gateway-openshift-default-696f47c6c-jrrjj\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.106057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.106022 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:44.518028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.518001 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj"] Nov 02 20:02:44.519635 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:02:44.519607 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5317a4_5400_4d8b_bdff_6a9697b040ad.slice/crio-945eb49c806e24241952a8c972581263458b46ec6a5b0824bc133cbe73a6658a WatchSource:0}: Error finding container 945eb49c806e24241952a8c972581263458b46ec6a5b0824bc133cbe73a6658a: Status 404 returned error can't find the container with id 945eb49c806e24241952a8c972581263458b46ec6a5b0824bc133cbe73a6658a Nov 02 20:02:44.521920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.521889 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:02:44.521983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.521957 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:02:44.522018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:44.521986 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:02:45.412470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:45.412434 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" event={"ID":"7b5317a4-5400-4d8b-bdff-6a9697b040ad","Type":"ContainerStarted","Data":"83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e"} Nov 02 20:02:45.412982 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:45.412488 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" event={"ID":"7b5317a4-5400-4d8b-bdff-6a9697b040ad","Type":"ContainerStarted","Data":"945eb49c806e24241952a8c972581263458b46ec6a5b0824bc133cbe73a6658a"} Nov 02 20:02:45.551577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:45.551514 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" podStartSLOduration=2.551499979 podStartE2EDuration="2.551499979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:45.549981288 +0000 UTC m=+608.160633379" watchObservedRunningTime="2026-11-02 20:02:45.551499979 +0000 UTC m=+608.162152069" Nov 02 20:02:46.107027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:46.106991 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:46.108512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:46.108490 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jrrjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:02:46.108601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:46.108535 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:02:47.106952 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:47.106912 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jrrjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:02:47.107394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:47.106967 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:02:48.107059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:48.107023 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jrrjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:02:48.107487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:48.107078 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:02:48.259880 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:02:48.259833 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:02:49.107156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:49.107119 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jrrjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:02:49.107675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:49.107184 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:02:50.107342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:50.107306 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jrrjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:02:50.107741 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:50.107376 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:02:51.106626 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:51.106585 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jrrjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:02:51.106810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:51.106653 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:02:52.807137 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:52.807104 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:52.807640 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:52.807500 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:02:52.808382 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:02:52.808357 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:03:18.236985 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:03:18.236903 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:03:48.236090 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:03:48.236051 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:03:57.823755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.823721 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh"] Nov 02 20:03:57.827470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.827450 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:57.876066 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.876031 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-cc64d3-openshift-default-dockercfg-dq8fc\"" Nov 02 20:03:57.924501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.924470 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh"] Nov 02 20:03:57.943711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.943683 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:57.943855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.943715 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:57.943855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.943745 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:57.943932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.943844 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8bzd\" (UniqueName: \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-kube-api-access-x8bzd\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:57.943932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.943899 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:57.944013 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.943993 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:57.944048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.944033 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:57.944083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.944067 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:57.944115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:57.944095 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d56f67-d429-4c97-9a92-1c8570de49f8-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.044744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.044706 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.044939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.044756 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.044939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.044787 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.044939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.044821 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d56f67-d429-4c97-9a92-1c8570de49f8-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.045121 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.044967 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.045121 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.045007 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.045121 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.045054 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.045121 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.045094 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8bzd\" (UniqueName: \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-kube-api-access-x8bzd\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.045121 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.045098 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.045430 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.045168 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.045430 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.045183 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.045539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.045461 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.045614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.045595 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.045674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.045639 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d56f67-d429-4c97-9a92-1c8570de49f8-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.047191 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.047167 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.047421 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.047399 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.108381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.108308 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8bzd\" (UniqueName: \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-kube-api-access-x8bzd\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.108621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.108603 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.140574 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.140544 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:58.207785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.206785 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn"] Nov 02 20:03:58.211658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.211632 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.260799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.260770 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-982b8a-openshift-default-dockercfg-lsv7m\"" Nov 02 20:03:58.266458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.266429 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn"] Nov 02 20:03:58.347879 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.347846 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.347879 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.347879 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.348128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.347963 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.348128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.348025 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-272gc\" (UniqueName: \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-kube-api-access-272gc\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.348128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.348045 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.348128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.348072 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.348305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.348222 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44434a4f-a23d-48fd-a593-b8aee99aaecc-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.348359 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.348297 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.348359 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.348349 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.449656 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.449568 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.449656 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.449600 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.449956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.449749 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.449956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.449813 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-272gc\" (UniqueName: \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-kube-api-access-272gc\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.449956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.449846 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.449956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.449875 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.449956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.449923 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44434a4f-a23d-48fd-a593-b8aee99aaecc-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.449956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.449956 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.450306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.449981 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.450306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.449996 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.450306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.450038 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.450478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.450420 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.450478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.450466 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.450706 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.450688 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44434a4f-a23d-48fd-a593-b8aee99aaecc-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.452231 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.452211 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.452473 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.452455 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.507498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.507464 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.507641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.507568 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-272gc\" (UniqueName: \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-kube-api-access-272gc\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.525426 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.525400 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:03:58.552690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.552660 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh"] Nov 02 20:03:58.554331 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:03:58.554303 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d56f67_d429_4c97_9a92_1c8570de49f8.slice/crio-ef4c6d4990f58410761aa5ed58f8f092a73be53b218d989278d7d65c9f132648 WatchSource:0}: Error finding container ef4c6d4990f58410761aa5ed58f8f092a73be53b218d989278d7d65c9f132648: Status 404 returned error can't find the container with id ef4c6d4990f58410761aa5ed58f8f092a73be53b218d989278d7d65c9f132648 Nov 02 20:03:58.556602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.556561 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:03:58.556770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.556650 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:03:58.556770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.556699 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:03:58.811993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.811957 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" event={"ID":"d5d56f67-d429-4c97-9a92-1c8570de49f8","Type":"ContainerStarted","Data":"57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727"} Nov 02 20:03:58.812160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.812019 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" event={"ID":"d5d56f67-d429-4c97-9a92-1c8570de49f8","Type":"ContainerStarted","Data":"ef4c6d4990f58410761aa5ed58f8f092a73be53b218d989278d7d65c9f132648"} Nov 02 20:03:58.904403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.904375 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn"] Nov 02 20:03:58.907548 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:03:58.907519 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44434a4f_a23d_48fd_a593_b8aee99aaecc.slice/crio-8aca8c2b2f22550252fee44ef0cea92b3ce863ea97947b8f7e396cc16a8eaa1e WatchSource:0}: Error finding container 8aca8c2b2f22550252fee44ef0cea92b3ce863ea97947b8f7e396cc16a8eaa1e: Status 404 returned error can't find the container with id 8aca8c2b2f22550252fee44ef0cea92b3ce863ea97947b8f7e396cc16a8eaa1e Nov 02 20:03:58.910441 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.910404 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:03:58.910520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.910487 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:03:58.910562 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.910534 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:03:58.920147 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:58.920091 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" podStartSLOduration=1.920074785 podStartE2EDuration="1.920074785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:58.919323946 +0000 UTC m=+681.529976039" watchObservedRunningTime="2026-11-02 20:03:58.920074785 +0000 UTC m=+681.530726877" Nov 02 20:03:59.141535 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:59.141173 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:59.146930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:59.146896 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:59.819226 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:59.819189 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" event={"ID":"44434a4f-a23d-48fd-a593-b8aee99aaecc","Type":"ContainerStarted","Data":"05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f"} Nov 02 20:03:59.819427 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:59.819251 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" event={"ID":"44434a4f-a23d-48fd-a593-b8aee99aaecc","Type":"ContainerStarted","Data":"8aca8c2b2f22550252fee44ef0cea92b3ce863ea97947b8f7e396cc16a8eaa1e"} Nov 02 20:03:59.819582 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:59.819562 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:59.820565 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:59.820546 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:03:59.930101 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:03:59.930050 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" podStartSLOduration=1.930034319 podStartE2EDuration="1.930034319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:59.927764605 +0000 UTC m=+682.538416698" watchObservedRunningTime="2026-11-02 20:03:59.930034319 +0000 UTC m=+682.540686410" Nov 02 20:04:00.526025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:00.525992 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:04:00.531402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:00.531375 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:04:00.824145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:00.824112 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:04:00.825116 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:00.825099 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:04:07.135118 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.135081 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq"] Nov 02 20:04:07.138937 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.138920 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:04:07.190417 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.190372 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9fcedd\"" Nov 02 20:04:07.236587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.236556 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-9fcedd-dockercfg-5j5tp\"" Nov 02 20:04:07.239250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.239220 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq"] Nov 02 20:04:07.243510 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.243485 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6xn\" (UniqueName: \"kubernetes.io/projected/e7f94892-433e-4a0e-8a6b-004aadfcab15-kube-api-access-sz6xn\") pod \"payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq\" (UID: \"e7f94892-433e-4a0e-8a6b-004aadfcab15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:04:07.243634 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.243615 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f94892-433e-4a0e-8a6b-004aadfcab15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq\" (UID: \"e7f94892-433e-4a0e-8a6b-004aadfcab15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:04:07.283029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.282997 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm"] Nov 02 20:04:07.286693 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.286677 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:04:07.338663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.338631 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm"] Nov 02 20:04:07.344408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.344381 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f94892-433e-4a0e-8a6b-004aadfcab15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq\" (UID: \"e7f94892-433e-4a0e-8a6b-004aadfcab15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:04:07.344557 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.344526 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6xn\" (UniqueName: \"kubernetes.io/projected/e7f94892-433e-4a0e-8a6b-004aadfcab15-kube-api-access-sz6xn\") pod \"payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq\" (UID: \"e7f94892-433e-4a0e-8a6b-004aadfcab15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:04:07.345095 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.345070 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f94892-433e-4a0e-8a6b-004aadfcab15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq\" (UID: \"e7f94892-433e-4a0e-8a6b-004aadfcab15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:04:07.399407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.399339 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6xn\" (UniqueName: \"kubernetes.io/projected/e7f94892-433e-4a0e-8a6b-004aadfcab15-kube-api-access-sz6xn\") pod \"payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq\" (UID: \"e7f94892-433e-4a0e-8a6b-004aadfcab15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:04:07.445869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.445837 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7s9d\" (UniqueName: \"kubernetes.io/projected/829f41dc-f27c-4d36-9670-eaebac12f098-kube-api-access-g7s9d\") pod \"payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm\" (UID: \"829f41dc-f27c-4d36-9670-eaebac12f098\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:04:07.446030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.445909 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829f41dc-f27c-4d36-9670-eaebac12f098-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm\" (UID: \"829f41dc-f27c-4d36-9670-eaebac12f098\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:04:07.451823 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.451801 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:04:07.547009 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.546977 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7s9d\" (UniqueName: \"kubernetes.io/projected/829f41dc-f27c-4d36-9670-eaebac12f098-kube-api-access-g7s9d\") pod \"payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm\" (UID: \"829f41dc-f27c-4d36-9670-eaebac12f098\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:04:07.547185 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.547057 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829f41dc-f27c-4d36-9670-eaebac12f098-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm\" (UID: \"829f41dc-f27c-4d36-9670-eaebac12f098\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:04:07.547766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.547746 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829f41dc-f27c-4d36-9670-eaebac12f098-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm\" (UID: \"829f41dc-f27c-4d36-9670-eaebac12f098\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:04:07.600045 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.600004 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7s9d\" (UniqueName: \"kubernetes.io/projected/829f41dc-f27c-4d36-9670-eaebac12f098-kube-api-access-g7s9d\") pod \"payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm\" (UID: \"829f41dc-f27c-4d36-9670-eaebac12f098\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:04:07.600227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.600211 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:04:07.812227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.812202 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq"] Nov 02 20:04:07.814826 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:04:07.814801 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f94892_433e_4a0e_8a6b_004aadfcab15.slice/crio-f67fcac02ed316dfb52ef48ce858acf821fe83f1e1a1fa4b3e07de8673ac10af WatchSource:0}: Error finding container f67fcac02ed316dfb52ef48ce858acf821fe83f1e1a1fa4b3e07de8673ac10af: Status 404 returned error can't find the container with id f67fcac02ed316dfb52ef48ce858acf821fe83f1e1a1fa4b3e07de8673ac10af Nov 02 20:04:07.816459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.816442 5399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:04:07.869397 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.869355 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" event={"ID":"e7f94892-433e-4a0e-8a6b-004aadfcab15","Type":"ContainerStarted","Data":"f67fcac02ed316dfb52ef48ce858acf821fe83f1e1a1fa4b3e07de8673ac10af"} Nov 02 20:04:07.969991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:07.969963 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm"] Nov 02 20:04:07.971731 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:04:07.971701 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829f41dc_f27c_4d36_9670_eaebac12f098.slice/crio-209e4a74b65e176d21e1e397863d2a917d56b5c6e49ef1cfef1a7a678668e368 WatchSource:0}: Error finding container 209e4a74b65e176d21e1e397863d2a917d56b5c6e49ef1cfef1a7a678668e368: Status 404 returned error can't find the container with id 209e4a74b65e176d21e1e397863d2a917d56b5c6e49ef1cfef1a7a678668e368 Nov 02 20:04:08.876261 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:08.876224 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" event={"ID":"829f41dc-f27c-4d36-9670-eaebac12f098","Type":"ContainerStarted","Data":"209e4a74b65e176d21e1e397863d2a917d56b5c6e49ef1cfef1a7a678668e368"} Nov 02 20:04:11.911396 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:11.911351 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" event={"ID":"829f41dc-f27c-4d36-9670-eaebac12f098","Type":"ContainerStarted","Data":"a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd"} Nov 02 20:04:11.911857 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:11.911422 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:04:11.912863 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:11.912841 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" event={"ID":"e7f94892-433e-4a0e-8a6b-004aadfcab15","Type":"ContainerStarted","Data":"a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef"} Nov 02 20:04:11.912974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:11.912892 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:04:12.020269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:12.020213 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" podStartSLOduration=1.993176798 podStartE2EDuration="5.020198847s" totalImagesPullingTime="3.027022049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:12.019033247 +0000 UTC m=+694.629685337" watchObservedRunningTime="2026-11-02 20:04:12.020198847 +0000 UTC m=+694.630850938" Nov 02 20:04:12.135459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:12.135400 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" podStartSLOduration=1.992249363 podStartE2EDuration="5.135380766s" totalImagesPullingTime="3.143131403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:12.134634536 +0000 UTC m=+694.745286628" watchObservedRunningTime="2026-11-02 20:04:12.135380766 +0000 UTC m=+694.746032858" Nov 02 20:04:14.700571 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.700530 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx"] Nov 02 20:04:14.705553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.705519 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:04:14.721713 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.721675 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0050aa91-905a-4fc3-835a-9c557c04ccdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx\" (UID: \"0050aa91-905a-4fc3-835a-9c557c04ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:04:14.721846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.721739 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsv5\" (UniqueName: \"kubernetes.io/projected/0050aa91-905a-4fc3-835a-9c557c04ccdd-kube-api-access-zfsv5\") pod \"payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx\" (UID: \"0050aa91-905a-4fc3-835a-9c557c04ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:04:14.755494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.755462 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-982b8a\"" Nov 02 20:04:14.757432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.757394 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx"] Nov 02 20:04:14.801332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.801301 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-982b8a-dockercfg-94njk\"" Nov 02 20:04:14.822890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.822853 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsv5\" (UniqueName: \"kubernetes.io/projected/0050aa91-905a-4fc3-835a-9c557c04ccdd-kube-api-access-zfsv5\") pod \"payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx\" (UID: \"0050aa91-905a-4fc3-835a-9c557c04ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:04:14.823092 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.822994 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0050aa91-905a-4fc3-835a-9c557c04ccdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx\" (UID: \"0050aa91-905a-4fc3-835a-9c557c04ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:04:14.823729 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.823709 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0050aa91-905a-4fc3-835a-9c557c04ccdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx\" (UID: \"0050aa91-905a-4fc3-835a-9c557c04ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:04:14.854050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.854012 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m"] Nov 02 20:04:14.857931 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.857909 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:04:14.879506 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.879475 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsv5\" (UniqueName: \"kubernetes.io/projected/0050aa91-905a-4fc3-835a-9c557c04ccdd-kube-api-access-zfsv5\") pod \"payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx\" (UID: \"0050aa91-905a-4fc3-835a-9c557c04ccdd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:04:14.911218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.911190 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m"] Nov 02 20:04:14.924806 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.924770 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6677808e-467a-4e97-be61-95b8f53df853-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m\" (UID: \"6677808e-467a-4e97-be61-95b8f53df853\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:04:14.924937 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:14.924817 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xb8\" (UniqueName: \"kubernetes.io/projected/6677808e-467a-4e97-be61-95b8f53df853-kube-api-access-82xb8\") pod \"payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m\" (UID: \"6677808e-467a-4e97-be61-95b8f53df853\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:04:15.026560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.026477 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6677808e-467a-4e97-be61-95b8f53df853-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m\" (UID: \"6677808e-467a-4e97-be61-95b8f53df853\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:04:15.026560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.026514 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xb8\" (UniqueName: \"kubernetes.io/projected/6677808e-467a-4e97-be61-95b8f53df853-kube-api-access-82xb8\") pod \"payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m\" (UID: \"6677808e-467a-4e97-be61-95b8f53df853\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:04:15.027227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.027205 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6677808e-467a-4e97-be61-95b8f53df853-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m\" (UID: \"6677808e-467a-4e97-be61-95b8f53df853\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:04:15.030253 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.030233 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:04:15.085317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.079810 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xb8\" (UniqueName: \"kubernetes.io/projected/6677808e-467a-4e97-be61-95b8f53df853-kube-api-access-82xb8\") pod \"payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m\" (UID: \"6677808e-467a-4e97-be61-95b8f53df853\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:04:15.171397 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.171365 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:04:15.393963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.393938 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx"] Nov 02 20:04:15.396158 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:04:15.396125 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0050aa91_905a_4fc3_835a_9c557c04ccdd.slice/crio-0487212f4af70ee37e97ea787684bebd358ff690da4ac45b654955ba92d01197 WatchSource:0}: Error finding container 0487212f4af70ee37e97ea787684bebd358ff690da4ac45b654955ba92d01197: Status 404 returned error can't find the container with id 0487212f4af70ee37e97ea787684bebd358ff690da4ac45b654955ba92d01197 Nov 02 20:04:15.540000 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.539975 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m"] Nov 02 20:04:15.541250 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:04:15.541223 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6677808e_467a_4e97_be61_95b8f53df853.slice/crio-685b577f25f6eb082662fb76f9aab6d3e88dd8245d6496b714f2a667363eddcd WatchSource:0}: Error finding container 685b577f25f6eb082662fb76f9aab6d3e88dd8245d6496b714f2a667363eddcd: Status 404 returned error can't find the container with id 685b577f25f6eb082662fb76f9aab6d3e88dd8245d6496b714f2a667363eddcd Nov 02 20:04:15.938971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.938930 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" event={"ID":"6677808e-467a-4e97-be61-95b8f53df853","Type":"ContainerStarted","Data":"1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e"} Nov 02 20:04:15.939485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.938989 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" event={"ID":"6677808e-467a-4e97-be61-95b8f53df853","Type":"ContainerStarted","Data":"685b577f25f6eb082662fb76f9aab6d3e88dd8245d6496b714f2a667363eddcd"} Nov 02 20:04:15.939485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.939113 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:04:15.940585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.940561 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" event={"ID":"0050aa91-905a-4fc3-835a-9c557c04ccdd","Type":"ContainerStarted","Data":"a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b"} Nov 02 20:04:15.940708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.940610 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" event={"ID":"0050aa91-905a-4fc3-835a-9c557c04ccdd","Type":"ContainerStarted","Data":"0487212f4af70ee37e97ea787684bebd358ff690da4ac45b654955ba92d01197"} Nov 02 20:04:15.940708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:15.940640 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:04:16.048082 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:16.048029 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" podStartSLOduration=2.048014258 podStartE2EDuration="2.048014258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:16.046563568 +0000 UTC m=+698.657215663" watchObservedRunningTime="2026-11-02 20:04:16.048014258 +0000 UTC m=+698.658666349" Nov 02 20:04:16.152607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:16.152556 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" podStartSLOduration=2.152541325 podStartE2EDuration="2.152541325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:16.150841746 +0000 UTC m=+698.761493838" watchObservedRunningTime="2026-11-02 20:04:16.152541325 +0000 UTC m=+698.763193416" Nov 02 20:04:17.254711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.254677 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc"] Nov 02 20:04:17.258482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.258463 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:04:17.307041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.307006 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-035e47\"" Nov 02 20:04:17.351212 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.351158 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90935866-4032-47ca-8938-c965d81fdda0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc\" (UID: \"90935866-4032-47ca-8938-c965d81fdda0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:04:17.352056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.351704 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q69jw\" (UniqueName: \"kubernetes.io/projected/90935866-4032-47ca-8938-c965d81fdda0-kube-api-access-q69jw\") pod \"payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc\" (UID: \"90935866-4032-47ca-8938-c965d81fdda0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:04:17.375443 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.375413 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-035e47-dockercfg-nr76n\"" Nov 02 20:04:17.380584 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.380547 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc"] Nov 02 20:04:17.419182 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.419151 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc"] Nov 02 20:04:17.422773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.422757 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:04:17.453311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.453261 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20aa971f-7e5c-4267-a7a8-b07b99676807-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc\" (UID: \"20aa971f-7e5c-4267-a7a8-b07b99676807\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:04:17.453461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.453355 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90935866-4032-47ca-8938-c965d81fdda0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc\" (UID: \"90935866-4032-47ca-8938-c965d81fdda0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:04:17.453461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.453437 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nwmf\" (UniqueName: \"kubernetes.io/projected/20aa971f-7e5c-4267-a7a8-b07b99676807-kube-api-access-8nwmf\") pod \"payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc\" (UID: \"20aa971f-7e5c-4267-a7a8-b07b99676807\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:04:17.453546 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.453500 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q69jw\" (UniqueName: \"kubernetes.io/projected/90935866-4032-47ca-8938-c965d81fdda0-kube-api-access-q69jw\") pod \"payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc\" (UID: \"90935866-4032-47ca-8938-c965d81fdda0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:04:17.454047 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.454031 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90935866-4032-47ca-8938-c965d81fdda0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc\" (UID: \"90935866-4032-47ca-8938-c965d81fdda0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:04:17.463447 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.463422 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc"] Nov 02 20:04:17.513366 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.513268 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q69jw\" (UniqueName: \"kubernetes.io/projected/90935866-4032-47ca-8938-c965d81fdda0-kube-api-access-q69jw\") pod \"payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc\" (UID: \"90935866-4032-47ca-8938-c965d81fdda0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:04:17.554903 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.554858 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20aa971f-7e5c-4267-a7a8-b07b99676807-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc\" (UID: \"20aa971f-7e5c-4267-a7a8-b07b99676807\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:04:17.555097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.554973 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nwmf\" (UniqueName: \"kubernetes.io/projected/20aa971f-7e5c-4267-a7a8-b07b99676807-kube-api-access-8nwmf\") pod \"payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc\" (UID: \"20aa971f-7e5c-4267-a7a8-b07b99676807\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:04:17.555626 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.555602 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20aa971f-7e5c-4267-a7a8-b07b99676807-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc\" (UID: \"20aa971f-7e5c-4267-a7a8-b07b99676807\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:04:17.572743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.572710 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:04:17.607507 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.607482 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nwmf\" (UniqueName: \"kubernetes.io/projected/20aa971f-7e5c-4267-a7a8-b07b99676807-kube-api-access-8nwmf\") pod \"payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc\" (UID: \"20aa971f-7e5c-4267-a7a8-b07b99676807\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:04:17.736338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.736300 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:04:17.937362 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.937333 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc"] Nov 02 20:04:17.939025 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:04:17.938997 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90935866_4032_47ca_8938_c965d81fdda0.slice/crio-d2b97eb520fb221b8866e55a0334b39ab297ffd96a8118822437f6f5552366a9 WatchSource:0}: Error finding container d2b97eb520fb221b8866e55a0334b39ab297ffd96a8118822437f6f5552366a9: Status 404 returned error can't find the container with id d2b97eb520fb221b8866e55a0334b39ab297ffd96a8118822437f6f5552366a9 Nov 02 20:04:17.951825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:17.951799 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" event={"ID":"90935866-4032-47ca-8938-c965d81fdda0","Type":"ContainerStarted","Data":"d2b97eb520fb221b8866e55a0334b39ab297ffd96a8118822437f6f5552366a9"} Nov 02 20:04:18.102025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:18.102001 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc"] Nov 02 20:04:18.104038 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:04:18.104004 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20aa971f_7e5c_4267_a7a8_b07b99676807.slice/crio-ef46fc9712a16efab26b0513b84cc381c2731f231ebeb2a4629b970ae05e9cd1 WatchSource:0}: Error finding container ef46fc9712a16efab26b0513b84cc381c2731f231ebeb2a4629b970ae05e9cd1: Status 404 returned error can't find the container with id ef46fc9712a16efab26b0513b84cc381c2731f231ebeb2a4629b970ae05e9cd1 Nov 02 20:04:18.238577 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:04:18.238540 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:04:18.958797 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:18.958757 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" event={"ID":"90935866-4032-47ca-8938-c965d81fdda0","Type":"ContainerStarted","Data":"07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04"} Nov 02 20:04:18.959245 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:18.958823 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:04:18.960313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:18.960286 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" event={"ID":"20aa971f-7e5c-4267-a7a8-b07b99676807","Type":"ContainerStarted","Data":"20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27"} Nov 02 20:04:18.960405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:18.960337 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" event={"ID":"20aa971f-7e5c-4267-a7a8-b07b99676807","Type":"ContainerStarted","Data":"ef46fc9712a16efab26b0513b84cc381c2731f231ebeb2a4629b970ae05e9cd1"} Nov 02 20:04:18.960405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:18.960359 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:04:19.064020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:19.063957 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" podStartSLOduration=2.063942664 podStartE2EDuration="2.063942664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:19.063239275 +0000 UTC m=+701.673891367" watchObservedRunningTime="2026-11-02 20:04:19.063942664 +0000 UTC m=+701.674594754" Nov 02 20:04:19.170949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:19.170892 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" podStartSLOduration=2.17087753 podStartE2EDuration="2.17087753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:19.168552629 +0000 UTC m=+701.779204721" watchObservedRunningTime="2026-11-02 20:04:19.17087753 +0000 UTC m=+701.781529712" Nov 02 20:04:19.663886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:19.663850 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:04:22.921629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:22.921597 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:04:22.922039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:22.921703 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:04:26.949404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:26.949369 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:04:26.949868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:26.949461 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:04:27.985663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:27.985618 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt"] Nov 02 20:04:27.989652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:27.989626 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.086512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.086474 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-2052e3-serving-cert\"" Nov 02 20:04:28.086767 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.086620 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:04:28.086767 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.086750 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 20:04:28.086949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.086662 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vqtwv\"" Nov 02 20:04:28.086949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.086620 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:04:28.089007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.088983 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt"] Nov 02 20:04:28.112870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.112835 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0930d956-8e72-4e36-bd77-5ff02a4892a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.113066 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.112920 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.113066 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.112987 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-metrics-certs\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.113175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.113081 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-977zd\" (UniqueName: \"kubernetes.io/projected/0930d956-8e72-4e36-bd77-5ff02a4892a2-kube-api-access-977zd\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.214092 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.214061 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-977zd\" (UniqueName: \"kubernetes.io/projected/0930d956-8e72-4e36-bd77-5ff02a4892a2-kube-api-access-977zd\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.214333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.214160 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0930d956-8e72-4e36-bd77-5ff02a4892a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.214333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.214182 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.214333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.214228 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-metrics-certs\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.214502 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:04:28.214356 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-serving-cert: secret "maas-api-e2e-worker-w3-2052e3-serving-cert" not found Nov 02 20:04:28.214502 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:04:28.214435 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-maas-api-tls podName:0930d956-8e72-4e36-bd77-5ff02a4892a2 nodeName:}" failed. No retries permitted until 2026-11-02 20:04:28.714404535 +0000 UTC m=+711.325056621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-maas-api-tls") pod "maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" (UID: "0930d956-8e72-4e36-bd77-5ff02a4892a2") : secret "maas-api-e2e-worker-w3-2052e3-serving-cert" not found Nov 02 20:04:28.215108 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.215079 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0930d956-8e72-4e36-bd77-5ff02a4892a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.216854 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.216832 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-metrics-certs\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.270549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.270469 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-977zd\" (UniqueName: \"kubernetes.io/projected/0930d956-8e72-4e36-bd77-5ff02a4892a2-kube-api-access-977zd\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.720366 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.720328 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.722933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.722907 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:28.907428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:28.907391 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:29.272369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:29.272338 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt"] Nov 02 20:04:29.274527 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:04:29.274502 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0930d956_8e72_4e36_bd77_5ff02a4892a2.slice/crio-e51b02c561323f0eed61469db8ad7c2a372bb817a285dd8d73bea55b1fbf6cba WatchSource:0}: Error finding container e51b02c561323f0eed61469db8ad7c2a372bb817a285dd8d73bea55b1fbf6cba: Status 404 returned error can't find the container with id e51b02c561323f0eed61469db8ad7c2a372bb817a285dd8d73bea55b1fbf6cba Nov 02 20:04:29.820836 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:29.820800 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:04:29.821032 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:29.820850 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:04:29.974000 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:29.973915 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:04:29.974156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:29.974075 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:04:30.020558 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.020524 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:04:30.037466 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.037435 5399 generic.go:356] "Generic (PLEG): container finished" podID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerID="57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727" exitCode=0 Nov 02 20:04:30.037627 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.037502 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" event={"ID":"d5d56f67-d429-4c97-9a92-1c8570de49f8","Type":"ContainerDied","Data":"57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727"} Nov 02 20:04:30.037890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.037874 5399 scope.go:122] "RemoveContainer" containerID="57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727" Nov 02 20:04:30.038780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.038760 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" event={"ID":"0930d956-8e72-4e36-bd77-5ff02a4892a2","Type":"ContainerStarted","Data":"e51b02c561323f0eed61469db8ad7c2a372bb817a285dd8d73bea55b1fbf6cba"} Nov 02 20:04:30.039219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.039197 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:30.039313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.039265 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:30.039380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.039328 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:30.040476 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.040455 5399 generic.go:356] "Generic (PLEG): container finished" podID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerID="05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f" exitCode=0 Nov 02 20:04:30.040559 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.040510 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" event={"ID":"44434a4f-a23d-48fd-a593-b8aee99aaecc","Type":"ContainerDied","Data":"05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f"} Nov 02 20:04:30.040856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.040840 5399 scope.go:122] "RemoveContainer" containerID="05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f" Nov 02 20:04:30.042234 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.042208 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:30.042298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.042270 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:30.042339 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.042316 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:30.824725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:30.824637 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:04:31.089725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:31.089640 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" event={"ID":"d5d56f67-d429-4c97-9a92-1c8570de49f8","Type":"ContainerStarted","Data":"2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f"} Nov 02 20:04:31.092239 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:31.092210 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" event={"ID":"44434a4f-a23d-48fd-a593-b8aee99aaecc","Type":"ContainerStarted","Data":"09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1"} Nov 02 20:04:31.159769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:31.159679 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:04:31.177323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:31.176466 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:04:31.177323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:31.176534 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:04:31.551355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:31.549674 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:04:31.560301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:31.559749 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 20:04:31.560301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:31.559813 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 20:04:32.099071 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:32.099040 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" event={"ID":"0930d956-8e72-4e36-bd77-5ff02a4892a2","Type":"ContainerStarted","Data":"1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf"} Nov 02 20:04:32.099580 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:32.099543 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:32.145307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:32.144690 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:04:32.145307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:32.144745 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:04:32.216754 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:32.216652 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" podStartSLOduration=2.516775705 podStartE2EDuration="5.216638497s" totalImagesPullingTime="2.699862792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:32.214457604 +0000 UTC m=+714.825109695" watchObservedRunningTime="2026-11-02 20:04:32.216638497 +0000 UTC m=+714.827290613" Nov 02 20:04:32.528007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:32.527501 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 20:04:32.528007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:32.527571 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 20:04:33.143227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:33.142737 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:04:33.143227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:33.142801 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:04:33.525858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:33.525775 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 20:04:33.525858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:33.525842 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 20:04:34.142748 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:34.142708 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:04:34.143076 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:34.143046 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:04:34.530352 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:34.526333 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 20:04:34.530352 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:34.526396 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 20:04:35.152300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:35.151384 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:04:35.152300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:35.151457 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:04:35.526076 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:35.525991 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 20:04:35.526076 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:35.526043 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 20:04:36.145301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:36.143698 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:04:36.145301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:36.143760 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:04:36.530538 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:36.530461 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:04:36.530863 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:36.530845 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:04:36.531507 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:36.531481 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:04:37.145136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:37.145104 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:04:37.145433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:37.145414 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:04:37.146201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:37.146186 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:04:39.118904 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:39.118876 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:04:41.839022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:41.838988 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:04:48.235839 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:04:48.235754 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:04:52.385236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:04:52.385198 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:05:09.068788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:09.068751 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:05:18.235682 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:05:18.235635 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:05:30.965010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:30.964974 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:05:35.657531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.657498 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk"] Nov 02 20:05:35.661060 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.661040 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.699622 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.699591 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-dshm\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.699772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.699649 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-home\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.699772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.699745 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-tmp-dir\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.699857 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.699788 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-model-cache\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.699919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.699900 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgqx\" (UniqueName: \"kubernetes.io/projected/d267cc3b-bbd7-42aa-b457-504072df67db-kube-api-access-rrgqx\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.699976 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.699950 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d267cc3b-bbd7-42aa-b457-504072df67db-tls-certs\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.759436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.759402 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-035e47\"/\"openshift-service-ca.crt\"" Nov 02 20:05:35.759754 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.759733 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035e47\"/\"default-dockercfg-sxn9n\"" Nov 02 20:05:35.759754 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.759737 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-035e47\"/\"kube-root-ca.crt\"" Nov 02 20:05:35.759893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.759802 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035e47\"/\"simulator-w0-035e47-kserve-self-signed-certs\"" Nov 02 20:05:35.761878 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.761859 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk"] Nov 02 20:05:35.800940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.800911 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgqx\" (UniqueName: \"kubernetes.io/projected/d267cc3b-bbd7-42aa-b457-504072df67db-kube-api-access-rrgqx\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.800940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.800950 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d267cc3b-bbd7-42aa-b457-504072df67db-tls-certs\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.801141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.800974 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-dshm\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.801141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.801015 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-home\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.801141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.801061 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-tmp-dir\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.801141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.801101 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-model-cache\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.801502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.801473 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-model-cache\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.802494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.802472 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-home\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.803033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.803002 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-tmp-dir\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.805437 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.805376 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d267cc3b-bbd7-42aa-b457-504072df67db-tls-certs\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.806095 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.806068 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-dshm\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.857332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.857299 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgqx\" (UniqueName: \"kubernetes.io/projected/d267cc3b-bbd7-42aa-b457-504072df67db-kube-api-access-rrgqx\") pod \"simulator-w0-035e47-kserve-7fc4cffb45-pqkpk\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:35.972825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:35.972746 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:36.341010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:36.340982 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk"] Nov 02 20:05:36.342671 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:05:36.342641 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd267cc3b_bbd7_42aa_b457_504072df67db.slice/crio-72d4b75faf5766e71567ef72e18347d26230f3e9e383523a04137c815a24b5dc WatchSource:0}: Error finding container 72d4b75faf5766e71567ef72e18347d26230f3e9e383523a04137c815a24b5dc: Status 404 returned error can't find the container with id 72d4b75faf5766e71567ef72e18347d26230f3e9e383523a04137c815a24b5dc Nov 02 20:05:36.474888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:36.474853 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" event={"ID":"d267cc3b-bbd7-42aa-b457-504072df67db","Type":"ContainerStarted","Data":"973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55"} Nov 02 20:05:36.475058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:36.474895 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" event={"ID":"d267cc3b-bbd7-42aa-b457-504072df67db","Type":"ContainerStarted","Data":"72d4b75faf5766e71567ef72e18347d26230f3e9e383523a04137c815a24b5dc"} Nov 02 20:05:36.582766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:36.582707 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podStartSLOduration=1.582691979 podStartE2EDuration="1.582691979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:36.580713784 +0000 UTC m=+779.191365874" watchObservedRunningTime="2026-11-02 20:05:36.582691979 +0000 UTC m=+779.193344073" Nov 02 20:05:36.973032 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:36.972999 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:42.124853 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.124816 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd"] Nov 02 20:05:42.128621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.128601 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.225579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.225514 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"simulator-w3-2052e3-kserve-self-signed-certs\"" Nov 02 20:05:42.225579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.225515 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"openshift-service-ca.crt\"" Nov 02 20:05:42.225579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.225515 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"kube-root-ca.crt\"" Nov 02 20:05:42.225881 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.225701 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2052e3\"/\"default-dockercfg-94w94\"" Nov 02 20:05:42.228216 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.228193 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd"] Nov 02 20:05:42.265403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.265368 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tmp-dir\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.265577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.265423 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjjwf\" (UniqueName: \"kubernetes.io/projected/3f1e6beb-1003-4bb8-bc07-ade27a80afce-kube-api-access-xjjwf\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.265577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.265539 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-home\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.265577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.265568 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-dshm\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.265699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.265608 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-model-cache\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.265699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.265640 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tls-certs\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.367099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.367058 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjjwf\" (UniqueName: \"kubernetes.io/projected/3f1e6beb-1003-4bb8-bc07-ade27a80afce-kube-api-access-xjjwf\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.367298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.367140 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-home\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.367298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.367177 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-dshm\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.367298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.367207 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-model-cache\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.367298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.367235 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tls-certs\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.367515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.367423 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tmp-dir\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.367593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.367572 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-home\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.367859 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.367840 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-model-cache\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.367957 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.367879 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tmp-dir\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.369363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.369338 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-dshm\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.369645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.369625 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tls-certs\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.420165 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.420085 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjjwf\" (UniqueName: \"kubernetes.io/projected/3f1e6beb-1003-4bb8-bc07-ade27a80afce-kube-api-access-xjjwf\") pod \"simulator-w3-2052e3-kserve-7cb544c977-wfjcd\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.439954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.439908 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:42.802495 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:42.802463 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd"] Nov 02 20:05:42.804117 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:05:42.804088 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f1e6beb_1003_4bb8_bc07_ade27a80afce.slice/crio-9125462a055409943087a0f8812b78391f33b45edd6c116b1e542de6f28bac1e WatchSource:0}: Error finding container 9125462a055409943087a0f8812b78391f33b45edd6c116b1e542de6f28bac1e: Status 404 returned error can't find the container with id 9125462a055409943087a0f8812b78391f33b45edd6c116b1e542de6f28bac1e Nov 02 20:05:43.538297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:43.537324 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" event={"ID":"3f1e6beb-1003-4bb8-bc07-ade27a80afce","Type":"ContainerStarted","Data":"2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad"} Nov 02 20:05:43.538297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:43.537381 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" event={"ID":"3f1e6beb-1003-4bb8-bc07-ade27a80afce","Type":"ContainerStarted","Data":"9125462a055409943087a0f8812b78391f33b45edd6c116b1e542de6f28bac1e"} Nov 02 20:05:43.645462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:43.645403 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podStartSLOduration=1.6453888349999999 podStartE2EDuration="1.645388835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:43.644265681 +0000 UTC m=+786.254917775" watchObservedRunningTime="2026-11-02 20:05:43.645388835 +0000 UTC m=+786.256040923" Nov 02 20:05:44.440541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:44.440503 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:45.973088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:45.973048 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:45.985188 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:45.985164 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:46.565872 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:46.565839 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:05:48.236000 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:05:48.235966 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:05:52.440901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:52.440864 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:52.453332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:52.453296 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:52.598573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:52.598542 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:05:53.133048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:05:53.133016 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:06:09.359341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:09.359303 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:06:18.238037 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:06:18.237947 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:06:26.300585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:26.300548 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:06:46.195896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.195863 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h"] Nov 02 20:06:46.199881 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.199852 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.216335 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.216307 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-home\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.216486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.216372 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb94p\" (UniqueName: \"kubernetes.io/projected/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-kube-api-access-rb94p\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.216486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.216464 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tls-certs\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.216599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.216528 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-dshm\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.216599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.216578 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-model-cache\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.216697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.216675 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tmp-dir\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.295485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.295449 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ae8e6e\"/\"openshift-service-ca.crt\"" Nov 02 20:06:46.295710 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.295472 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ae8e6e\"/\"default-dockercfg-6lc96\"" Nov 02 20:06:46.295710 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.295503 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ae8e6e\"/\"premium-w5-ae8e6e-kserve-self-signed-certs\"" Nov 02 20:06:46.296147 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.296128 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ae8e6e\"/\"kube-root-ca.crt\"" Nov 02 20:06:46.297922 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.297901 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h"] Nov 02 20:06:46.317979 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.317954 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb94p\" (UniqueName: \"kubernetes.io/projected/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-kube-api-access-rb94p\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.317979 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.317982 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tls-certs\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.318207 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.318013 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-dshm\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.318207 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.318061 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-model-cache\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.318207 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.318133 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tmp-dir\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.318411 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.318256 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-home\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.318582 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.318554 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-model-cache\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.318582 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.318579 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tmp-dir\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.318761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.318610 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-home\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.320433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.320410 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-dshm\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.320800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.320776 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tls-certs\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.371362 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.371327 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb94p\" (UniqueName: \"kubernetes.io/projected/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-kube-api-access-rb94p\") pod \"premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.512482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.512397 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:46.891653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:46.891621 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h"] Nov 02 20:06:46.893531 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:06:46.893503 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8e7ae1b_8a2c_4fa1_90d2_42c6df2309a6.slice/crio-c1596d4afd2ba4de83df0700f7c109b3852e2bd0d4dc56bc9690965f9a8073f5 WatchSource:0}: Error finding container c1596d4afd2ba4de83df0700f7c109b3852e2bd0d4dc56bc9690965f9a8073f5: Status 404 returned error can't find the container with id c1596d4afd2ba4de83df0700f7c109b3852e2bd0d4dc56bc9690965f9a8073f5 Nov 02 20:06:47.892902 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:47.892867 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" event={"ID":"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6","Type":"ContainerStarted","Data":"d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb"} Nov 02 20:06:47.892902 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:47.892908 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" event={"ID":"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6","Type":"ContainerStarted","Data":"c1596d4afd2ba4de83df0700f7c109b3852e2bd0d4dc56bc9690965f9a8073f5"} Nov 02 20:06:47.998771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:47.998516 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podStartSLOduration=1.998497312 podStartE2EDuration="1.998497312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:47.998062699 +0000 UTC m=+850.608714793" watchObservedRunningTime="2026-11-02 20:06:47.998497312 +0000 UTC m=+850.609149437" Nov 02 20:06:48.237049 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:06:48.236951 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:06:48.512949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:48.512862 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:53.102102 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:53.102065 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:06:56.512518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:56.512483 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:56.525395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:56.525370 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:56.960014 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:56.959978 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:06:59.041842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.041804 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk"] Nov 02 20:06:59.045858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.045838 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.141425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.141389 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035e47\"/\"premium-w0-035e47-kserve-self-signed-certs\"" Nov 02 20:06:59.143984 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.143959 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk"] Nov 02 20:06:59.153501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.153393 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-dshm\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.153501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.153423 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-model-cache\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.153590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.153577 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765d761d-8567-46a2-a441-c185a232f060-tls-certs\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.153624 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.153603 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-tmp-dir\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.153657 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.153625 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvlq\" (UniqueName: \"kubernetes.io/projected/765d761d-8567-46a2-a441-c185a232f060-kube-api-access-crvlq\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.153753 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.153735 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-home\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.255270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.255235 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765d761d-8567-46a2-a441-c185a232f060-tls-certs\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.255270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.255286 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-tmp-dir\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.255502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.255403 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvlq\" (UniqueName: \"kubernetes.io/projected/765d761d-8567-46a2-a441-c185a232f060-kube-api-access-crvlq\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.255502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.255444 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-home\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.255614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.255596 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-dshm\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.255669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.255629 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-tmp-dir\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.255669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.255642 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-model-cache\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.255841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.255818 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-home\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.256013 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.255993 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-model-cache\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.257678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.257650 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-dshm\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.258831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.258808 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765d761d-8567-46a2-a441-c185a232f060-tls-certs\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.308176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.308144 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvlq\" (UniqueName: \"kubernetes.io/projected/765d761d-8567-46a2-a441-c185a232f060-kube-api-access-crvlq\") pod \"premium-w0-035e47-kserve-9f46d69fb-c2fvk\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.358334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.358298 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:06:59.729504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.729476 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk"] Nov 02 20:06:59.730189 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:06:59.730158 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod765d761d_8567_46a2_a441_c185a232f060.slice/crio-2131eda9c5a0785cc6214756eff3e03c5456b00d64811e586786590f163e9b67 WatchSource:0}: Error finding container 2131eda9c5a0785cc6214756eff3e03c5456b00d64811e586786590f163e9b67: Status 404 returned error can't find the container with id 2131eda9c5a0785cc6214756eff3e03c5456b00d64811e586786590f163e9b67 Nov 02 20:06:59.969544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.969504 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" event={"ID":"765d761d-8567-46a2-a441-c185a232f060","Type":"ContainerStarted","Data":"5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f"} Nov 02 20:06:59.969721 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:06:59.969560 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" event={"ID":"765d761d-8567-46a2-a441-c185a232f060","Type":"ContainerStarted","Data":"2131eda9c5a0785cc6214756eff3e03c5456b00d64811e586786590f163e9b67"} Nov 02 20:07:00.075616 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:00.075511 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podStartSLOduration=1.07549307 podStartE2EDuration="1.07549307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:00.075241829 +0000 UTC m=+862.685893944" watchObservedRunningTime="2026-11-02 20:07:00.07549307 +0000 UTC m=+862.686145162" Nov 02 20:07:00.358474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:00.358386 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:07:09.359512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:09.359473 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:07:09.372040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:09.372014 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:07:10.039504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:10.039475 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:07:18.236462 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:07:18.236422 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:07:19.295156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.295121 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f"] Nov 02 20:07:19.299254 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.299232 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.370951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.370921 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-dshm\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.371120 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.370962 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-home\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.371176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.371134 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdbz\" (UniqueName: \"kubernetes.io/projected/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-kube-api-access-5sdbz\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.371218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.371175 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tmp-dir\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.371257 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.371245 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tls-certs\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.371365 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.371351 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-model-cache\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.394864 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.394837 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"premium-w1-accdf5-kserve-self-signed-certs\"" Nov 02 20:07:19.394864 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.394841 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"default-dockercfg-sqlwk\"" Nov 02 20:07:19.395078 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.394880 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"openshift-service-ca.crt\"" Nov 02 20:07:19.395078 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.394909 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"kube-root-ca.crt\"" Nov 02 20:07:19.397760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.397734 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f"] Nov 02 20:07:19.472661 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.472626 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdbz\" (UniqueName: \"kubernetes.io/projected/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-kube-api-access-5sdbz\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.472661 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.472665 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tmp-dir\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.472929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.472687 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tls-certs\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.472929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.472751 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-model-cache\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.472929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.472806 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-dshm\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.472929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.472847 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-home\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.473112 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.473091 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tmp-dir\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.473186 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.473164 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-model-cache\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.473266 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.473242 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-home\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.475233 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.475206 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-dshm\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.475456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.475440 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tls-certs\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.525339 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.525265 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdbz\" (UniqueName: \"kubernetes.io/projected/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-kube-api-access-5sdbz\") pod \"premium-w1-accdf5-kserve-85488bb77f-6nc7f\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.612131 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.612098 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:19.979765 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:19.979740 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f"] Nov 02 20:07:19.981347 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:07:19.981316 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a27fd9a_920b_4562_8c2c_57c9cf8ffe59.slice/crio-2f4a5b175f9ac3bd0b2f13907bebc03bc038c0be8045721b84c45a9ae0d5647c WatchSource:0}: Error finding container 2f4a5b175f9ac3bd0b2f13907bebc03bc038c0be8045721b84c45a9ae0d5647c: Status 404 returned error can't find the container with id 2f4a5b175f9ac3bd0b2f13907bebc03bc038c0be8045721b84c45a9ae0d5647c Nov 02 20:07:20.086506 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:20.086471 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" event={"ID":"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59","Type":"ContainerStarted","Data":"484cc73fd665feb190b05e4bc410c014306a77a1c56cd1b9071c5262e4cbaaa8"} Nov 02 20:07:20.086506 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:20.086513 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" event={"ID":"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59","Type":"ContainerStarted","Data":"2f4a5b175f9ac3bd0b2f13907bebc03bc038c0be8045721b84c45a9ae0d5647c"} Nov 02 20:07:20.193977 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:20.193919 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podStartSLOduration=1.193904846 podStartE2EDuration="1.193904846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:20.192783059 +0000 UTC m=+882.803435152" watchObservedRunningTime="2026-11-02 20:07:20.193904846 +0000 UTC m=+882.804556936" Nov 02 20:07:20.613084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:20.613055 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:29.613026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:29.612994 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:29.625560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:29.625539 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:30.173354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:30.173322 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:07:30.421027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:07:30.420996 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:07:48.237548 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:07:48.237458 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:08:00.617566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.617530 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7"] Nov 02 20:08:00.621570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.621548 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.670531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.670503 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-035e47-openshift-default-dockercfg-rx8rw\"" Nov 02 20:08:00.719974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.719933 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7"] Nov 02 20:08:00.731020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.730986 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cff83a-d81e-49bb-b7e1-956e820bc615-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.731172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.731049 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.731172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.731149 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.731255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.731188 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.731314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.731259 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.731394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.731374 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.731439 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.731415 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm27q\" (UniqueName: \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-kube-api-access-gm27q\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.731535 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.731518 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.731662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.731644 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833163 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833129 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833383 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833194 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cff83a-d81e-49bb-b7e1-956e820bc615-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833383 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833227 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833383 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833247 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833383 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833267 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833504 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833586 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833589 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833644 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm27q\" (UniqueName: \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-kube-api-access-gm27q\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833663 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833724 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833833 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.833987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.833968 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cff83a-d81e-49bb-b7e1-956e820bc615-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.834107 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.834085 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.835966 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.835943 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.836067 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.835964 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.886514 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.886440 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.886637 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.886556 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm27q\" (UniqueName: \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-kube-api-access-gm27q\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:00.935138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:00.935105 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:01.349458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:01.349427 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7"] Nov 02 20:08:01.350773 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:08:01.350746 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cff83a_d81e_49bb_b7e1_956e820bc615.slice/crio-cdc2e2e865faf52327bc5ae27f7fabdaa59674db964bfb1d470a9c8c77b00850 WatchSource:0}: Error finding container cdc2e2e865faf52327bc5ae27f7fabdaa59674db964bfb1d470a9c8c77b00850: Status 404 returned error can't find the container with id cdc2e2e865faf52327bc5ae27f7fabdaa59674db964bfb1d470a9c8c77b00850 Nov 02 20:08:01.353019 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:01.352987 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:08:01.353087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:01.353054 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:08:01.353087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:01.353081 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:08:02.359824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:02.359770 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" event={"ID":"43cff83a-d81e-49bb-b7e1-956e820bc615","Type":"ContainerStarted","Data":"c6c23cb8694700a9c19628b1bd83f8dc928eb2cb23b1321bc22eedaa2c3d7076"} Nov 02 20:08:02.360296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:02.359847 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" event={"ID":"43cff83a-d81e-49bb-b7e1-956e820bc615","Type":"ContainerStarted","Data":"cdc2e2e865faf52327bc5ae27f7fabdaa59674db964bfb1d470a9c8c77b00850"} Nov 02 20:08:02.562767 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:02.562710 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" podStartSLOduration=2.5626942059999998 podStartE2EDuration="2.562694206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:02.513801938 +0000 UTC m=+925.124454031" watchObservedRunningTime="2026-11-02 20:08:02.562694206 +0000 UTC m=+925.173346348" Nov 02 20:08:02.947375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:02.947338 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:02.951414 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:02.951379 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:08:02.951586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:02.951448 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" podUID="43cff83a-d81e-49bb-b7e1-956e820bc615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:08:03.951302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:03.950418 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:08:03.951302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:03.950503 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" podUID="43cff83a-d81e-49bb-b7e1-956e820bc615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:08:04.951303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:04.947748 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:08:04.951303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:04.947823 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" podUID="43cff83a-d81e-49bb-b7e1-956e820bc615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:08:05.889873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:05.889838 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:08:05.951304 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:05.950847 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:08:05.951304 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:05.950912 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" podUID="43cff83a-d81e-49bb-b7e1-956e820bc615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:08:06.945300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:06.944375 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:08:06.945300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:06.944440 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" podUID="43cff83a-d81e-49bb-b7e1-956e820bc615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:08:08.116218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:08.116179 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:08.116678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:08.116633 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:08.117372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:08.117352 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:08:18.236890 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:08:18.236845 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:08:41.081071 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:41.081036 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:08:47.355440 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.355392 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r"] Nov 02 20:08:47.362523 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.361351 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:47.412102 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.412062 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r"] Nov 02 20:08:47.454190 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.454151 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:08:47.454380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.454241 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:08:47.469725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.469686 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8sl2\" (UniqueName: \"kubernetes.io/projected/a37f4181-9977-4ac5-a86e-3ae635912d09-kube-api-access-g8sl2\") pod \"maas-api-revoke-sub-429d3fa5-zr28r\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:47.469899 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.469852 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-429d3fa5-zr28r\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:47.469899 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.469895 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-429d3fa5-zr28r\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:47.570541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.570505 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-429d3fa5-zr28r\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:47.570735 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.570618 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8sl2\" (UniqueName: \"kubernetes.io/projected/a37f4181-9977-4ac5-a86e-3ae635912d09-kube-api-access-g8sl2\") pod \"maas-api-revoke-sub-429d3fa5-zr28r\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:47.570735 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.570691 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-429d3fa5-zr28r\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:47.571354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.571330 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-429d3fa5-zr28r\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:47.572967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.572938 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-429d3fa5-zr28r\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:47.623398 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.623328 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8sl2\" (UniqueName: \"kubernetes.io/projected/a37f4181-9977-4ac5-a86e-3ae635912d09-kube-api-access-g8sl2\") pod \"maas-api-revoke-sub-429d3fa5-zr28r\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:47.677758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:47.677729 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:48.040746 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:48.040715 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r"] Nov 02 20:08:48.042335 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:08:48.042309 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37f4181_9977_4ac5_a86e_3ae635912d09.slice/crio-7efc61cdf0e4f5bacfd5bd0eb50365f2c171b8ef3080d8af8b8728683ebe7690 WatchSource:0}: Error finding container 7efc61cdf0e4f5bacfd5bd0eb50365f2c171b8ef3080d8af8b8728683ebe7690: Status 404 returned error can't find the container with id 7efc61cdf0e4f5bacfd5bd0eb50365f2c171b8ef3080d8af8b8728683ebe7690 Nov 02 20:08:48.240050 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:08:48.240014 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:08:48.643215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:48.643124 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" event={"ID":"a37f4181-9977-4ac5-a86e-3ae635912d09","Type":"ContainerStarted","Data":"85e37e18e78b096676790e658f2006c73b84bda700f717bbe81e9fe9e69378f4"} Nov 02 20:08:48.643215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:48.643193 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" event={"ID":"a37f4181-9977-4ac5-a86e-3ae635912d09","Type":"ContainerStarted","Data":"7efc61cdf0e4f5bacfd5bd0eb50365f2c171b8ef3080d8af8b8728683ebe7690"} Nov 02 20:08:48.761712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:48.761657 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" podStartSLOduration=1.761643091 podStartE2EDuration="1.761643091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:48.759965683 +0000 UTC m=+971.370617776" watchObservedRunningTime="2026-11-02 20:08:48.761643091 +0000 UTC m=+971.372295181" Nov 02 20:08:49.650181 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:49.650137 5399 generic.go:356] "Generic (PLEG): container finished" podID="a37f4181-9977-4ac5-a86e-3ae635912d09" containerID="85e37e18e78b096676790e658f2006c73b84bda700f717bbe81e9fe9e69378f4" exitCode=0 Nov 02 20:08:49.650647 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:49.650227 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" event={"ID":"a37f4181-9977-4ac5-a86e-3ae635912d09","Type":"ContainerDied","Data":"85e37e18e78b096676790e658f2006c73b84bda700f717bbe81e9fe9e69378f4"} Nov 02 20:08:50.804848 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:50.804818 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:50.906542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:50.906507 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-xks-ca\") pod \"a37f4181-9977-4ac5-a86e-3ae635912d09\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " Nov 02 20:08:50.906744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:50.906652 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-openshift-ca\") pod \"a37f4181-9977-4ac5-a86e-3ae635912d09\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " Nov 02 20:08:50.906744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:50.906715 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37f4181-9977-4ac5-a86e-3ae635912d09-kube-api-access-g8sl2\" (UniqueName: \"kubernetes.io/projected/a37f4181-9977-4ac5-a86e-3ae635912d09-kube-api-access-g8sl2\") pod \"a37f4181-9977-4ac5-a86e-3ae635912d09\" (UID: \"a37f4181-9977-4ac5-a86e-3ae635912d09\") " Nov 02 20:08:50.907023 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:50.906991 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-openshift-ca" pod "a37f4181-9977-4ac5-a86e-3ae635912d09" (UID: "a37f4181-9977-4ac5-a86e-3ae635912d09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:50.908688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:50.908668 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-xks-ca" pod "a37f4181-9977-4ac5-a86e-3ae635912d09" (UID: "a37f4181-9977-4ac5-a86e-3ae635912d09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:08:50.908828 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:50.908804 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37f4181-9977-4ac5-a86e-3ae635912d09-kube-api-access-g8sl2" pod "a37f4181-9977-4ac5-a86e-3ae635912d09" (UID: "a37f4181-9977-4ac5-a86e-3ae635912d09"). InnerVolumeSpecName "kube-api-access-g8sl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:51.008091 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:51.008051 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:51.008091 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:51.008082 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a37f4181-9977-4ac5-a86e-3ae635912d09-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:51.008091 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:51.008092 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8sl2\" (UniqueName: \"kubernetes.io/projected/a37f4181-9977-4ac5-a86e-3ae635912d09-kube-api-access-g8sl2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:51.662877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:51.662835 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" event={"ID":"a37f4181-9977-4ac5-a86e-3ae635912d09","Type":"ContainerDied","Data":"7efc61cdf0e4f5bacfd5bd0eb50365f2c171b8ef3080d8af8b8728683ebe7690"} Nov 02 20:08:51.662877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:51.662862 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r" Nov 02 20:08:51.663074 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:08:51.662898 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7efc61cdf0e4f5bacfd5bd0eb50365f2c171b8ef3080d8af8b8728683ebe7690" Nov 02 20:09:04.178335 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.178298 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq"] Nov 02 20:09:04.179041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.179018 5399 cpu_manager.go:414] "containerMap: removing container" podUID="a37f4181-9977-4ac5-a86e-3ae635912d09" containerName="revoke-keys" Nov 02 20:09:04.179140 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.179042 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37f4181-9977-4ac5-a86e-3ae635912d09" containerName="revoke-keys" Nov 02 20:09:04.179197 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.179166 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37f4181-9977-4ac5-a86e-3ae635912d09" containerName="revoke-keys" Nov 02 20:09:04.182798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.182776 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.249533 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.249495 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrtvr\" (UniqueName: \"kubernetes.io/projected/a011e758-72da-4b69-825d-3d5998bfd021-kube-api-access-jrtvr\") pod \"maas-api-revoke-sub-78373750-jlbpq\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.249697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.249611 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78373750-jlbpq\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.249697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.249679 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78373750-jlbpq\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.277925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.277886 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:09:04.277925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.277912 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:09:04.280555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.280527 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq"] Nov 02 20:09:04.350354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.350324 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtvr\" (UniqueName: \"kubernetes.io/projected/a011e758-72da-4b69-825d-3d5998bfd021-kube-api-access-jrtvr\") pod \"maas-api-revoke-sub-78373750-jlbpq\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.350535 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.350386 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78373750-jlbpq\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.350535 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.350424 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78373750-jlbpq\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.351038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.351019 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78373750-jlbpq\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.352706 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.352686 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78373750-jlbpq\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.403514 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.403481 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtvr\" (UniqueName: \"kubernetes.io/projected/a011e758-72da-4b69-825d-3d5998bfd021-kube-api-access-jrtvr\") pod \"maas-api-revoke-sub-78373750-jlbpq\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.496719 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.496644 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:04.857513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:04.857484 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq"] Nov 02 20:09:04.858703 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:09:04.858678 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda011e758_72da_4b69_825d_3d5998bfd021.slice/crio-e30e53d5a891d637f0eecb2588715222c126d8907ee8d498a930ac9e017d211c WatchSource:0}: Error finding container e30e53d5a891d637f0eecb2588715222c126d8907ee8d498a930ac9e017d211c: Status 404 returned error can't find the container with id e30e53d5a891d637f0eecb2588715222c126d8907ee8d498a930ac9e017d211c Nov 02 20:09:05.744993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:05.744965 5399 generic.go:356] "Generic (PLEG): container finished" podID="a011e758-72da-4b69-825d-3d5998bfd021" containerID="b672231b17fb1d918ea5c347f7a5c142444b3cf8a8fda10d14c49e753dfcbcb4" exitCode=0 Nov 02 20:09:05.745342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:05.745048 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" event={"ID":"a011e758-72da-4b69-825d-3d5998bfd021","Type":"ContainerDied","Data":"b672231b17fb1d918ea5c347f7a5c142444b3cf8a8fda10d14c49e753dfcbcb4"} Nov 02 20:09:05.745342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:05.745100 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" event={"ID":"a011e758-72da-4b69-825d-3d5998bfd021","Type":"ContainerStarted","Data":"e30e53d5a891d637f0eecb2588715222c126d8907ee8d498a930ac9e017d211c"} Nov 02 20:09:06.897065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:06.897040 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:06.978934 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:06.978904 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-xks-ca\") pod \"a011e758-72da-4b69-825d-3d5998bfd021\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " Nov 02 20:09:06.979108 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:06.979026 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-openshift-ca\") pod \"a011e758-72da-4b69-825d-3d5998bfd021\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " Nov 02 20:09:06.979176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:06.979147 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a011e758-72da-4b69-825d-3d5998bfd021-kube-api-access-jrtvr\" (UniqueName: \"kubernetes.io/projected/a011e758-72da-4b69-825d-3d5998bfd021-kube-api-access-jrtvr\") pod \"a011e758-72da-4b69-825d-3d5998bfd021\" (UID: \"a011e758-72da-4b69-825d-3d5998bfd021\") " Nov 02 20:09:06.979424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:06.979394 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-openshift-ca" pod "a011e758-72da-4b69-825d-3d5998bfd021" (UID: "a011e758-72da-4b69-825d-3d5998bfd021"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:06.979557 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:06.979543 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:06.981050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:06.981030 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-xks-ca" pod "a011e758-72da-4b69-825d-3d5998bfd021" (UID: "a011e758-72da-4b69-825d-3d5998bfd021"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:09:06.981141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:06.981101 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a011e758-72da-4b69-825d-3d5998bfd021-kube-api-access-jrtvr" pod "a011e758-72da-4b69-825d-3d5998bfd021" (UID: "a011e758-72da-4b69-825d-3d5998bfd021"). InnerVolumeSpecName "kube-api-access-jrtvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:07.080161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:07.080127 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrtvr\" (UniqueName: \"kubernetes.io/projected/a011e758-72da-4b69-825d-3d5998bfd021-kube-api-access-jrtvr\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:07.080161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:07.080155 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a011e758-72da-4b69-825d-3d5998bfd021-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:07.759957 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:07.759926 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" Nov 02 20:09:07.760606 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:07.759916 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq" event={"ID":"a011e758-72da-4b69-825d-3d5998bfd021","Type":"ContainerDied","Data":"e30e53d5a891d637f0eecb2588715222c126d8907ee8d498a930ac9e017d211c"} Nov 02 20:09:07.760818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:07.760626 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e30e53d5a891d637f0eecb2588715222c126d8907ee8d498a930ac9e017d211c" Nov 02 20:09:16.468253 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.468211 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb"] Nov 02 20:09:16.468752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.468742 5399 cpu_manager.go:414] "containerMap: removing container" podUID="a011e758-72da-4b69-825d-3d5998bfd021" containerName="revoke-keys" Nov 02 20:09:16.468793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.468754 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a011e758-72da-4b69-825d-3d5998bfd021" containerName="revoke-keys" Nov 02 20:09:16.468858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.468846 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a011e758-72da-4b69-825d-3d5998bfd021" containerName="revoke-keys" Nov 02 20:09:16.472465 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.472450 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.523336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.523299 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2052e3-openshift-default-dockercfg-8r7gl\"" Nov 02 20:09:16.572200 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.572164 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb"] Nov 02 20:09:16.579919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.579885 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.580084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.579928 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.580084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.580021 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad575b7-b339-4c48-8119-c971790bbacb-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.580084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.580055 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.580211 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.580166 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad575b7-b339-4c48-8119-c971790bbacb-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.580265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.580218 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.580413 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.580389 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.580479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.580429 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.580658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.580626 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcpmq\" (UniqueName: \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-kube-api-access-lcpmq\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682245 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad575b7-b339-4c48-8119-c971790bbacb-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682315 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682341 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682362 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682423 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpmq\" (UniqueName: \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-kube-api-access-lcpmq\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682487 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682518 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682552 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad575b7-b339-4c48-8119-c971790bbacb-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682583 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682899 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.682969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.682943 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.683022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.683006 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.683081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.683051 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.683137 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.683095 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad575b7-b339-4c48-8119-c971790bbacb-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.684738 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.684710 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.684974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.684957 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad575b7-b339-4c48-8119-c971790bbacb-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.734981 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.734908 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.735115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.734990 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpmq\" (UniqueName: \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-kube-api-access-lcpmq\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:16.785693 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:16.785660 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:17.202564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:17.202531 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb"] Nov 02 20:09:17.237692 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:09:17.237661 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad575b7_b339_4c48_8119_c971790bbacb.slice/crio-10c057bb14a6bf7e87dc3b3dda3ef33ea86860ae179355b580ff9c8ba3c00c06 WatchSource:0}: Error finding container 10c057bb14a6bf7e87dc3b3dda3ef33ea86860ae179355b580ff9c8ba3c00c06: Status 404 returned error can't find the container with id 10c057bb14a6bf7e87dc3b3dda3ef33ea86860ae179355b580ff9c8ba3c00c06 Nov 02 20:09:17.240228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:17.240193 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:17.240354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:17.240258 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:17.240354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:17.240316 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:17.853818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:17.853778 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" event={"ID":"1ad575b7-b339-4c48-8119-c971790bbacb","Type":"ContainerStarted","Data":"44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84"} Nov 02 20:09:17.854250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:17.853841 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" event={"ID":"1ad575b7-b339-4c48-8119-c971790bbacb","Type":"ContainerStarted","Data":"10c057bb14a6bf7e87dc3b3dda3ef33ea86860ae179355b580ff9c8ba3c00c06"} Nov 02 20:09:17.969211 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:17.969146 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" podStartSLOduration=1.9691318519999998 podStartE2EDuration="1.969131852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:17.966653025 +0000 UTC m=+1000.577305117" watchObservedRunningTime="2026-11-02 20:09:17.969131852 +0000 UTC m=+1000.579783950" Nov 02 20:09:18.261998 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:09:18.261907 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:09:18.786242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:18.786211 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:18.787761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:18.787740 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:09:18.787838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:18.787819 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:09:19.786394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:19.786362 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:09:19.786775 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:19.786430 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:09:20.786765 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:20.786731 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:09:20.787268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:20.786798 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:09:21.454793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.454752 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5"] Nov 02 20:09:21.458888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.458862 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.510147 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.510116 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5"] Nov 02 20:09:21.553300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.552483 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:09:21.553300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.552540 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:09:21.558676 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.558638 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79c10ee9-mdhb5\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.558839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.558711 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79c10ee9-mdhb5\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.558996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.558970 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllj5\" (UniqueName: \"kubernetes.io/projected/bb9c8142-0f46-4599-9aa0-31a2d62100ec-kube-api-access-qllj5\") pod \"maas-api-revoke-sub-79c10ee9-mdhb5\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.659761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.659715 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllj5\" (UniqueName: \"kubernetes.io/projected/bb9c8142-0f46-4599-9aa0-31a2d62100ec-kube-api-access-qllj5\") pod \"maas-api-revoke-sub-79c10ee9-mdhb5\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.659935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.659796 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79c10ee9-mdhb5\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.659935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.659839 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79c10ee9-mdhb5\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.660767 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.660743 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79c10ee9-mdhb5\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.662299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.662269 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79c10ee9-mdhb5\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.712952 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.712884 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllj5\" (UniqueName: \"kubernetes.io/projected/bb9c8142-0f46-4599-9aa0-31a2d62100ec-kube-api-access-qllj5\") pod \"maas-api-revoke-sub-79c10ee9-mdhb5\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.776851 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.776815 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:21.786461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.786426 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:09:21.786601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:21.786495 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:09:22.161552 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:22.161522 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5"] Nov 02 20:09:22.162443 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:09:22.162418 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9c8142_0f46_4599_9aa0_31a2d62100ec.slice/crio-d9853c649f923a2bfd1c086b3a1bf9aa074eb1d9a19fd2e70451c7c8fdf74ea1 WatchSource:0}: Error finding container d9853c649f923a2bfd1c086b3a1bf9aa074eb1d9a19fd2e70451c7c8fdf74ea1: Status 404 returned error can't find the container with id d9853c649f923a2bfd1c086b3a1bf9aa074eb1d9a19fd2e70451c7c8fdf74ea1 Nov 02 20:09:22.786538 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:22.786455 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:09:22.786538 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:22.786513 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:09:22.888746 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:22.888707 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" event={"ID":"bb9c8142-0f46-4599-9aa0-31a2d62100ec","Type":"ContainerStarted","Data":"9e7f6e93008c3b27297325d52a4233258a3b01824ba1ad9c56bd00f847d45995"} Nov 02 20:09:22.888939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:22.888761 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" event={"ID":"bb9c8142-0f46-4599-9aa0-31a2d62100ec","Type":"ContainerStarted","Data":"d9853c649f923a2bfd1c086b3a1bf9aa074eb1d9a19fd2e70451c7c8fdf74ea1"} Nov 02 20:09:23.006754 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:23.006697 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" podStartSLOduration=2.006681762 podStartE2EDuration="2.006681762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:23.005242152 +0000 UTC m=+1005.615894246" watchObservedRunningTime="2026-11-02 20:09:23.006681762 +0000 UTC m=+1005.617333859" Nov 02 20:09:23.786059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:23.786019 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:09:23.786522 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:23.786083 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:09:23.897220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:23.897178 5399 generic.go:356] "Generic (PLEG): container finished" podID="bb9c8142-0f46-4599-9aa0-31a2d62100ec" containerID="9e7f6e93008c3b27297325d52a4233258a3b01824ba1ad9c56bd00f847d45995" exitCode=0 Nov 02 20:09:23.897423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:23.897296 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" event={"ID":"bb9c8142-0f46-4599-9aa0-31a2d62100ec","Type":"ContainerDied","Data":"9e7f6e93008c3b27297325d52a4233258a3b01824ba1ad9c56bd00f847d45995"} Nov 02 20:09:24.790361 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:24.790334 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:24.790803 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:24.790698 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:24.791538 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:24.791518 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:09:25.054199 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.054132 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:25.206775 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.206742 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9c8142-0f46-4599-9aa0-31a2d62100ec-kube-api-access-qllj5\" (UniqueName: \"kubernetes.io/projected/bb9c8142-0f46-4599-9aa0-31a2d62100ec-kube-api-access-qllj5\") pod \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " Nov 02 20:09:25.206940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.206923 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-openshift-ca\") pod \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " Nov 02 20:09:25.207029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.206997 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-xks-ca\") pod \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\" (UID: \"bb9c8142-0f46-4599-9aa0-31a2d62100ec\") " Nov 02 20:09:25.207290 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.207245 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-openshift-ca" pod "bb9c8142-0f46-4599-9aa0-31a2d62100ec" (UID: "bb9c8142-0f46-4599-9aa0-31a2d62100ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:25.207464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.207443 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:25.208903 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.208885 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9c8142-0f46-4599-9aa0-31a2d62100ec-kube-api-access-qllj5" pod "bb9c8142-0f46-4599-9aa0-31a2d62100ec" (UID: "bb9c8142-0f46-4599-9aa0-31a2d62100ec"). InnerVolumeSpecName "kube-api-access-qllj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:25.209097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.209076 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-xks-ca" pod "bb9c8142-0f46-4599-9aa0-31a2d62100ec" (UID: "bb9c8142-0f46-4599-9aa0-31a2d62100ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:09:25.308856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.308831 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qllj5\" (UniqueName: \"kubernetes.io/projected/bb9c8142-0f46-4599-9aa0-31a2d62100ec-kube-api-access-qllj5\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:25.309019 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.308858 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb9c8142-0f46-4599-9aa0-31a2d62100ec-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:25.426907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.426869 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:09:25.912359 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.912325 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" Nov 02 20:09:25.912359 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.912346 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5" event={"ID":"bb9c8142-0f46-4599-9aa0-31a2d62100ec","Type":"ContainerDied","Data":"d9853c649f923a2bfd1c086b3a1bf9aa074eb1d9a19fd2e70451c7c8fdf74ea1"} Nov 02 20:09:25.912945 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:25.912390 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9853c649f923a2bfd1c086b3a1bf9aa074eb1d9a19fd2e70451c7c8fdf74ea1" Nov 02 20:09:29.971046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:29.971011 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98"] Nov 02 20:09:29.973644 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:29.971675 5399 cpu_manager.go:414] "containerMap: removing container" podUID="bb9c8142-0f46-4599-9aa0-31a2d62100ec" containerName="revoke-keys" Nov 02 20:09:29.973644 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:29.971695 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9c8142-0f46-4599-9aa0-31a2d62100ec" containerName="revoke-keys" Nov 02 20:09:29.973644 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:29.971838 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9c8142-0f46-4599-9aa0-31a2d62100ec" containerName="revoke-keys" Nov 02 20:09:29.975208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:29.975189 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.063862 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.063816 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.064065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.063888 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5498af7b-e63c-4dee-ac23-6e48f373243e-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.064065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.063991 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.064065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.064024 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.064231 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.064056 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.064231 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.064163 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg6xt\" (UniqueName: \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-kube-api-access-gg6xt\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.064231 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.064216 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.064407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.064318 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.064407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.064363 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.070042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.070016 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ae8e6e-openshift-default-dockercfg-4p7q9\"" Nov 02 20:09:30.073314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.073098 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98"] Nov 02 20:09:30.165095 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165057 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.165293 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165114 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.165293 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165158 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.165293 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165198 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg6xt\" (UniqueName: \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-kube-api-access-gg6xt\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.165293 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165243 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.165528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165306 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.165528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165345 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.165528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165377 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.165528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165436 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5498af7b-e63c-4dee-ac23-6e48f373243e-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.165738 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165596 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.165972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.165943 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.166111 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.166082 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.166191 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.166111 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5498af7b-e63c-4dee-ac23-6e48f373243e-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.166259 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.166241 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.167769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.167746 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.168055 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.168037 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.218725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.218687 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.218725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.218726 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg6xt\" (UniqueName: \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-kube-api-access-gg6xt\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.288051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.287974 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:30.701573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.701538 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98"] Nov 02 20:09:30.702694 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:09:30.702670 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5498af7b_e63c_4dee_ac23_6e48f373243e.slice/crio-e9fe419ca147d841ae9b47f683f3f96e2067a6a65f8566e7cc58a68456a5bc6b WatchSource:0}: Error finding container e9fe419ca147d841ae9b47f683f3f96e2067a6a65f8566e7cc58a68456a5bc6b: Status 404 returned error can't find the container with id e9fe419ca147d841ae9b47f683f3f96e2067a6a65f8566e7cc58a68456a5bc6b Nov 02 20:09:30.705013 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.704984 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:30.705088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.705045 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:30.705088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.705074 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:30.943708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.943670 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" event={"ID":"5498af7b-e63c-4dee-ac23-6e48f373243e","Type":"ContainerStarted","Data":"585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147"} Nov 02 20:09:30.943886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.943724 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" event={"ID":"5498af7b-e63c-4dee-ac23-6e48f373243e","Type":"ContainerStarted","Data":"e9fe419ca147d841ae9b47f683f3f96e2067a6a65f8566e7cc58a68456a5bc6b"} Nov 02 20:09:30.969378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.969299 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl"] Nov 02 20:09:30.973292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:30.973265 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.024928 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.024896 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-accdf5-openshift-default-dockercfg-c8k9m\"" Nov 02 20:09:31.028501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.028464 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl"] Nov 02 20:09:31.056770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.056714 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" podStartSLOduration=2.056699497 podStartE2EDuration="2.056699497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:31.053003445 +0000 UTC m=+1013.663655535" watchObservedRunningTime="2026-11-02 20:09:31.056699497 +0000 UTC m=+1013.667351588" Nov 02 20:09:31.077379 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.077347 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.077546 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.077401 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.077546 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.077483 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.077678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.077650 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.077785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.077707 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.077785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.077770 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhslw\" (UniqueName: \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-kube-api-access-jhslw\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.077866 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.077813 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.077866 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.077849 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.077935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.077867 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.178547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.178519 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.178723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.178554 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.178723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.178625 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.178723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.178647 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.178723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.178666 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhslw\" (UniqueName: \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-kube-api-access-jhslw\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.178723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.178693 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.178996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.178733 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.178996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.178827 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.178996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.178887 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.179150 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.179007 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.179150 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.179044 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.179248 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.179199 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.179310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.179241 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.179544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.179519 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.181122 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.181103 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.181360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.181341 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.231762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.231682 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.231890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.231860 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhslw\" (UniqueName: \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-kube-api-access-jhslw\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.287021 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.286991 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:31.288657 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.288626 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:31.290001 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.289978 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:09:31.290119 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.290021 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:09:31.709401 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.709375 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl"] Nov 02 20:09:31.710636 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:09:31.710611 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439a19f0_c499_4ef7_a96c_ce8f6ee07df8.slice/crio-c2332957471fe5948dc3b692f3626d9896405f23702af85036d0ce64c4e4a4c7 WatchSource:0}: Error finding container c2332957471fe5948dc3b692f3626d9896405f23702af85036d0ce64c4e4a4c7: Status 404 returned error can't find the container with id c2332957471fe5948dc3b692f3626d9896405f23702af85036d0ce64c4e4a4c7 Nov 02 20:09:31.713050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.713018 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:31.713146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.713081 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:31.713146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.713110 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:31.963561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.963468 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" event={"ID":"439a19f0-c499-4ef7-a96c-ce8f6ee07df8","Type":"ContainerStarted","Data":"3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478"} Nov 02 20:09:31.963561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:31.963532 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" event={"ID":"439a19f0-c499-4ef7-a96c-ce8f6ee07df8","Type":"ContainerStarted","Data":"c2332957471fe5948dc3b692f3626d9896405f23702af85036d0ce64c4e4a4c7"} Nov 02 20:09:32.072503 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:32.072455 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" podStartSLOduration=2.072441454 podStartE2EDuration="2.072441454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:32.070807567 +0000 UTC m=+1014.681459659" watchObservedRunningTime="2026-11-02 20:09:32.072441454 +0000 UTC m=+1014.683093544" Nov 02 20:09:32.293147 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:32.293053 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:32.293928 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:32.293895 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:09:32.294071 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:32.293952 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:09:32.294141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:32.293897 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:09:32.294207 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:32.294139 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:09:33.288206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:33.288171 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:09:33.288653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:33.288222 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:09:33.288653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:33.288322 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:09:33.288653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:33.288368 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:09:33.949305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:33.947080 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt"] Nov 02 20:09:33.967300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:33.965604 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:33.995708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:33.995566 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt"] Nov 02 20:09:34.014653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.014619 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqhpf\" (UniqueName: \"kubernetes.io/projected/c7f79054-8e32-49e9-9865-a00889f49002-kube-api-access-nqhpf\") pod \"maas-api-revoke-sub-e379a6bf-ngbjt\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:34.014810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.014723 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e379a6bf-ngbjt\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:34.014810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.014751 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e379a6bf-ngbjt\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:34.052584 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.052550 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:09:34.080388 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.054894 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:09:34.116393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.116348 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e379a6bf-ngbjt\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:34.116677 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.116659 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e379a6bf-ngbjt\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:34.116953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.116932 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqhpf\" (UniqueName: \"kubernetes.io/projected/c7f79054-8e32-49e9-9865-a00889f49002-kube-api-access-nqhpf\") pod \"maas-api-revoke-sub-e379a6bf-ngbjt\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:34.117137 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.117006 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e379a6bf-ngbjt\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:34.121325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.119004 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e379a6bf-ngbjt\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:34.179287 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.179235 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqhpf\" (UniqueName: \"kubernetes.io/projected/c7f79054-8e32-49e9-9865-a00889f49002-kube-api-access-nqhpf\") pod \"maas-api-revoke-sub-e379a6bf-ngbjt\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:34.288204 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.288073 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:09:34.288204 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.288141 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:09:34.288764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.288370 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:09:34.288764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.288410 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:09:34.297593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.297566 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:34.730597 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:34.727169 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt"] Nov 02 20:09:35.008345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:35.008309 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" event={"ID":"c7f79054-8e32-49e9-9865-a00889f49002","Type":"ContainerStarted","Data":"9d648fff926bbfbba5e8dd970ba769a227f7e1480eca02fd6a9f9ff50b371b88"} Nov 02 20:09:35.288383 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:35.288256 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:09:35.288383 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:35.288341 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:09:35.289349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:35.289313 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:09:35.289467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:35.289356 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:09:36.017892 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:36.017853 5399 generic.go:356] "Generic (PLEG): container finished" podID="c7f79054-8e32-49e9-9865-a00889f49002" containerID="d222259858e7ba8862e0e0e98dae7af2367dba701b1d0954f3e8be085af43a41" exitCode=0 Nov 02 20:09:36.018062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:36.017902 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" event={"ID":"c7f79054-8e32-49e9-9865-a00889f49002","Type":"ContainerDied","Data":"d222259858e7ba8862e0e0e98dae7af2367dba701b1d0954f3e8be085af43a41"} Nov 02 20:09:36.288143 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:36.288046 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:09:36.288143 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:36.288103 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:09:36.291022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:36.290749 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:09:36.291022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:36.290831 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:09:37.303048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.292164 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:09:37.303048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.292265 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:09:37.303048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.302120 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:09:37.303048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.302196 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:09:37.338504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.338464 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:37.459692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.459654 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-xks-ca\") pod \"c7f79054-8e32-49e9-9865-a00889f49002\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " Nov 02 20:09:37.460090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.460061 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f79054-8e32-49e9-9865-a00889f49002-kube-api-access-nqhpf\" (UniqueName: \"kubernetes.io/projected/c7f79054-8e32-49e9-9865-a00889f49002-kube-api-access-nqhpf\") pod \"c7f79054-8e32-49e9-9865-a00889f49002\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " Nov 02 20:09:37.460314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.460295 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-openshift-ca\") pod \"c7f79054-8e32-49e9-9865-a00889f49002\" (UID: \"c7f79054-8e32-49e9-9865-a00889f49002\") " Nov 02 20:09:37.461254 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.461229 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-openshift-ca" pod "c7f79054-8e32-49e9-9865-a00889f49002" (UID: "c7f79054-8e32-49e9-9865-a00889f49002"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:37.464205 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.464171 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-xks-ca" pod "c7f79054-8e32-49e9-9865-a00889f49002" (UID: "c7f79054-8e32-49e9-9865-a00889f49002"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:09:37.476471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.476436 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f79054-8e32-49e9-9865-a00889f49002-kube-api-access-nqhpf" pod "c7f79054-8e32-49e9-9865-a00889f49002" (UID: "c7f79054-8e32-49e9-9865-a00889f49002"). InnerVolumeSpecName "kube-api-access-nqhpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:37.577951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.564688 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqhpf\" (UniqueName: \"kubernetes.io/projected/c7f79054-8e32-49e9-9865-a00889f49002-kube-api-access-nqhpf\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:37.577951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.577403 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:37.577951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:37.577419 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f79054-8e32-49e9-9865-a00889f49002-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:38.037196 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:38.037105 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" event={"ID":"c7f79054-8e32-49e9-9865-a00889f49002","Type":"ContainerDied","Data":"9d648fff926bbfbba5e8dd970ba769a227f7e1480eca02fd6a9f9ff50b371b88"} Nov 02 20:09:38.037640 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:38.037620 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d648fff926bbfbba5e8dd970ba769a227f7e1480eca02fd6a9f9ff50b371b88" Nov 02 20:09:38.037781 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:38.037511 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt" Nov 02 20:09:38.292234 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:38.292143 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 02 20:09:38.292486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:38.292451 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 02 20:09:38.297502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:38.297475 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:38.297936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:38.297920 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:38.298903 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:38.298878 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:09:39.292502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:39.292474 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:39.292938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:39.292872 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:39.293847 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:39.293815 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:09:46.799238 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:46.799200 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj"] Nov 02 20:09:46.799725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:46.799448 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerName="istio-proxy" containerID="cri-o://83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e" gracePeriod=30 Nov 02 20:09:48.238401 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:09:48.238365 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:09:52.068797 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.068733 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:09:52.139079 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.139042 5399 generic.go:356] "Generic (PLEG): container finished" podID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerID="83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e" exitCode=0 Nov 02 20:09:52.139250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.139091 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" Nov 02 20:09:52.139250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.139136 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" event={"ID":"7b5317a4-5400-4d8b-bdff-6a9697b040ad","Type":"ContainerDied","Data":"83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e"} Nov 02 20:09:52.139250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.139199 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj" event={"ID":"7b5317a4-5400-4d8b-bdff-6a9697b040ad","Type":"ContainerDied","Data":"945eb49c806e24241952a8c972581263458b46ec6a5b0824bc133cbe73a6658a"} Nov 02 20:09:52.139446 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.139229 5399 scope.go:122] "RemoveContainer" containerID="83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e" Nov 02 20:09:52.148887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.148872 5399 scope.go:122] "RemoveContainer" containerID="83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e" Nov 02 20:09:52.149156 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:09:52.149137 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e\": container with ID starting with 83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e not found: ID does not exist" containerID="83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e" Nov 02 20:09:52.149207 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.149165 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e"} err="failed to get container status \"83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e\": rpc error: code = NotFound desc = could not find container \"83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e\": container with ID starting with 83b1d4d0b76a5d21b33a677bd5d76dbd1e36fab43dbfff73cdd92c16fbbc7d2e not found: ID does not exist" Nov 02 20:09:52.208490 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208458 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istiod-ca-cert\") pod \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " Nov 02 20:09:52.208658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208503 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-kube-api-access-mjqtg\" (UniqueName: \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-kube-api-access-mjqtg\") pod \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " Nov 02 20:09:52.208658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208575 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-data\") pod \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " Nov 02 20:09:52.208658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208628 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-token\" (UniqueName: \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-token\") pod \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " Nov 02 20:09:52.208816 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208695 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-credential-socket\") pod \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " Nov 02 20:09:52.208868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208856 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-socket\") pod \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " Nov 02 20:09:52.208922 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208890 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-certs\") pod \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " Nov 02 20:09:52.208974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208820 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istiod-ca-cert" pod "7b5317a4-5400-4d8b-bdff-6a9697b040ad" (UID: "7b5317a4-5400-4d8b-bdff-6a9697b040ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:52.208974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208919 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-data" pod "7b5317a4-5400-4d8b-bdff-6a9697b040ad" (UID: "7b5317a4-5400-4d8b-bdff-6a9697b040ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:52.208974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208949 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-envoy\") pod \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " Nov 02 20:09:52.209135 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.208976 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-credential-socket" pod "7b5317a4-5400-4d8b-bdff-6a9697b040ad" (UID: "7b5317a4-5400-4d8b-bdff-6a9697b040ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:52.209135 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.209035 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-podinfo\") pod \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\" (UID: \"7b5317a4-5400-4d8b-bdff-6a9697b040ad\") " Nov 02 20:09:52.209135 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.209120 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-certs" pod "7b5317a4-5400-4d8b-bdff-6a9697b040ad" (UID: "7b5317a4-5400-4d8b-bdff-6a9697b040ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:52.209333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.209140 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-socket" pod "7b5317a4-5400-4d8b-bdff-6a9697b040ad" (UID: "7b5317a4-5400-4d8b-bdff-6a9697b040ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:52.209639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.209585 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:52.209639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.209614 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:52.209639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.209631 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:52.209822 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.209646 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:52.209822 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.209660 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:52.211054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.211033 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-envoy" pod "7b5317a4-5400-4d8b-bdff-6a9697b040ad" (UID: "7b5317a4-5400-4d8b-bdff-6a9697b040ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:52.211154 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.211030 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-token" pod "7b5317a4-5400-4d8b-bdff-6a9697b040ad" (UID: "7b5317a4-5400-4d8b-bdff-6a9697b040ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:52.211215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.211167 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-kube-api-access-mjqtg" pod "7b5317a4-5400-4d8b-bdff-6a9697b040ad" (UID: "7b5317a4-5400-4d8b-bdff-6a9697b040ad"). InnerVolumeSpecName "kube-api-access-mjqtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:52.211314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.211294 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-podinfo" pod "7b5317a4-5400-4d8b-bdff-6a9697b040ad" (UID: "7b5317a4-5400-4d8b-bdff-6a9697b040ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:09:52.310793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.310762 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:52.310793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.310788 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:52.310793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.310800 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5317a4-5400-4d8b-bdff-6a9697b040ad-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:52.311026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.310809 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjqtg\" (UniqueName: \"kubernetes.io/projected/7b5317a4-5400-4d8b-bdff-6a9697b040ad-kube-api-access-mjqtg\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:52.555192 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.555157 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj"] Nov 02 20:09:52.561738 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:52.561712 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jrrjj"] Nov 02 20:09:53.507592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:53.507560 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" path="/var/lib/kubelet/pods/7b5317a4-5400-4d8b-bdff-6a9697b040ad/volumes" Nov 02 20:09:56.474341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:09:56.474302 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:10:18.237737 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:10:18.237697 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:10:31.701038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:31.700999 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:10:48.238133 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:10:48.238035 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:10:52.858842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.858805 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q"] Nov 02 20:10:52.859338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.859323 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c7f79054-8e32-49e9-9865-a00889f49002" containerName="revoke-keys" Nov 02 20:10:52.859394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.859339 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f79054-8e32-49e9-9865-a00889f49002" containerName="revoke-keys" Nov 02 20:10:52.859394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.859376 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerName="istio-proxy" Nov 02 20:10:52.859394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.859382 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerName="istio-proxy" Nov 02 20:10:52.859493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.859466 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5317a4-5400-4d8b-bdff-6a9697b040ad" containerName="istio-proxy" Nov 02 20:10:52.859493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.859477 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f79054-8e32-49e9-9865-a00889f49002" containerName="revoke-keys" Nov 02 20:10:52.864170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.864151 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:52.903467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.903434 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-home\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:52.903617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.903541 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5krnk\" (UniqueName: \"kubernetes.io/projected/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-kube-api-access-5krnk\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:52.903659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.903626 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-model-cache\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:52.903774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.903757 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tmp-dir\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:52.903814 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.903796 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-dshm\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:52.903864 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.903812 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tls-certs\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:52.919095 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.919036 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q"] Nov 02 20:10:52.959752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.959711 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"kube-root-ca.crt\"" Nov 02 20:10:52.959752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.959719 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"openshift-service-ca.crt\"" Nov 02 20:10:52.959956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.959772 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"default-dockercfg-j9ncr\"" Nov 02 20:10:52.959956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:52.959853 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc64d3\"/\"distinct-2-w4-cc64d3-kserve-self-signed-certs\"" Nov 02 20:10:53.004477 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.004446 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tmp-dir\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.004652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.004488 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-dshm\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.004652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.004503 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tls-certs\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.004652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.004555 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-home\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.004652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.004575 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5krnk\" (UniqueName: \"kubernetes.io/projected/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-kube-api-access-5krnk\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.004652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.004605 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-model-cache\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.004917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.004895 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tmp-dir\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.005042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.005023 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-home\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.005080 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.005043 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-model-cache\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.006941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.006887 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-dshm\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.007243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.007222 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tls-certs\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.057625 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.057592 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5krnk\" (UniqueName: \"kubernetes.io/projected/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-kube-api-access-5krnk\") pod \"distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.176134 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.176050 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:10:53.544191 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:53.544165 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q"] Nov 02 20:10:53.545554 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:10:53.545530 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0e04ae7_2c61_4595_8e07_2a7349bb01b9.slice/crio-2553887b1790328cd1a7e537f73f24c733b5bb3d47a5e354e33d1756acf368e2 WatchSource:0}: Error finding container 2553887b1790328cd1a7e537f73f24c733b5bb3d47a5e354e33d1756acf368e2: Status 404 returned error can't find the container with id 2553887b1790328cd1a7e537f73f24c733b5bb3d47a5e354e33d1756acf368e2 Nov 02 20:10:54.534886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:54.534851 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" event={"ID":"d0e04ae7-2c61-4595-8e07-2a7349bb01b9","Type":"ContainerStarted","Data":"ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f"} Nov 02 20:10:54.535291 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:54.534897 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" event={"ID":"d0e04ae7-2c61-4595-8e07-2a7349bb01b9","Type":"ContainerStarted","Data":"2553887b1790328cd1a7e537f73f24c733b5bb3d47a5e354e33d1756acf368e2"} Nov 02 20:10:54.642459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:54.642404 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podStartSLOduration=2.642389509 podStartE2EDuration="2.642389509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:54.641520554 +0000 UTC m=+1097.252172647" watchObservedRunningTime="2026-11-02 20:10:54.642389509 +0000 UTC m=+1097.253041619" Nov 02 20:10:55.176938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:10:55.176907 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:11:01.663961 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:01.663926 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:11:03.177115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:03.177077 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:11:03.189850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:03.189824 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:11:03.606707 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:03.606678 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:11:09.345895 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.345850 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb"] Nov 02 20:11:09.349951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.349926 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.445167 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.445129 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-accdf5\"/\"distinct-w1-accdf5-kserve-self-signed-certs\"" Nov 02 20:11:09.448107 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.448083 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb"] Nov 02 20:11:09.496045 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.495991 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-home\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.496296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.496160 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-model-cache\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.496446 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.496307 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-dshm\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.496446 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.496413 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/201c4192-e93b-428e-9695-c3c6d9585d83-tls-certs\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.496601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.496499 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-tmp-dir\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.496601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.496559 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqk5x\" (UniqueName: \"kubernetes.io/projected/201c4192-e93b-428e-9695-c3c6d9585d83-kube-api-access-gqk5x\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.597555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.597472 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-home\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.597555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.597512 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-model-cache\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.597778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.597574 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-dshm\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.597778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.597617 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/201c4192-e93b-428e-9695-c3c6d9585d83-tls-certs\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.597778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.597663 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-tmp-dir\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.597778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.597726 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqk5x\" (UniqueName: \"kubernetes.io/projected/201c4192-e93b-428e-9695-c3c6d9585d83-kube-api-access-gqk5x\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.597987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.597881 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-home\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.597987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.597936 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-model-cache\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.598089 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.598017 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-tmp-dir\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.600222 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.600195 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-dshm\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.600406 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.600388 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/201c4192-e93b-428e-9695-c3c6d9585d83-tls-certs\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.651663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.651634 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqk5x\" (UniqueName: \"kubernetes.io/projected/201c4192-e93b-428e-9695-c3c6d9585d83-kube-api-access-gqk5x\") pod \"distinct-w1-accdf5-kserve-7d948c9d94-7qktb\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:09.662544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:09.662516 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:10.034548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:10.034516 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb"] Nov 02 20:11:10.035184 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:11:10.035158 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod201c4192_e93b_428e_9695_c3c6d9585d83.slice/crio-d625f676c311327ab7d2d6a4fc4fcaba3b8f4ec3428ae0a6d650024f02c9c1f4 WatchSource:0}: Error finding container d625f676c311327ab7d2d6a4fc4fcaba3b8f4ec3428ae0a6d650024f02c9c1f4: Status 404 returned error can't find the container with id d625f676c311327ab7d2d6a4fc4fcaba3b8f4ec3428ae0a6d650024f02c9c1f4 Nov 02 20:11:10.639950 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:10.639917 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" event={"ID":"201c4192-e93b-428e-9695-c3c6d9585d83","Type":"ContainerStarted","Data":"9899cd035cc91d4583726b8c592c97376af81389da191ad0cdddc0b2b4e36b44"} Nov 02 20:11:10.640421 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:10.639958 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" event={"ID":"201c4192-e93b-428e-9695-c3c6d9585d83","Type":"ContainerStarted","Data":"d625f676c311327ab7d2d6a4fc4fcaba3b8f4ec3428ae0a6d650024f02c9c1f4"} Nov 02 20:11:10.663059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:10.663028 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:10.747595 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:10.747539 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podStartSLOduration=1.747523656 podStartE2EDuration="1.747523656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:10.745778978 +0000 UTC m=+1113.356431079" watchObservedRunningTime="2026-11-02 20:11:10.747523656 +0000 UTC m=+1113.358175747" Nov 02 20:11:18.236070 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:11:18.236025 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:11:19.663433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:19.663395 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:19.676890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:19.676860 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:19.706225 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:19.706193 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:11:42.931404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:42.931373 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:11:47.249782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.249746 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk"] Nov 02 20:11:47.255011 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.254994 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.320813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.320781 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk"] Nov 02 20:11:47.331444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.331412 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.331604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.331520 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.331604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.331575 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.331709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.331604 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.331709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.331696 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwpb\" (UniqueName: \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-kube-api-access-jpwpb\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.331799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.331716 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.331799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.331737 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.331799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.331755 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.331900 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.331869 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.433376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433266 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.433550 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433475 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.433679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433600 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.433679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433610 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.433679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433661 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.434175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433738 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwpb\" (UniqueName: \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-kube-api-access-jpwpb\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.434175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433766 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.434175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433804 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.434175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433835 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.434175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433916 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.434175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.433921 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.434175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.434070 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.434474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.434254 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.434474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.434390 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.436206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.436184 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.436375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.436358 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.487462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.487426 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.487640 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.487616 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwpb\" (UniqueName: \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-kube-api-access-jpwpb\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.569022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.568991 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:47.997900 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:47.997875 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk"] Nov 02 20:11:47.998287 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:11:47.998250 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e242ab5_a0fc_47c2_ace5_ad38ab49b0ab.slice/crio-e697831be0262ca5015706fa8106f05ee08b9119a7fe864b231253fc883d14cc WatchSource:0}: Error finding container e697831be0262ca5015706fa8106f05ee08b9119a7fe864b231253fc883d14cc: Status 404 returned error can't find the container with id e697831be0262ca5015706fa8106f05ee08b9119a7fe864b231253fc883d14cc Nov 02 20:11:48.000593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:48.000550 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:11:48.000701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:48.000645 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:11:48.000701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:48.000689 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:11:48.237429 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:11:48.237384 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:11:48.880213 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:48.880171 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" event={"ID":"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab","Type":"ContainerStarted","Data":"b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d"} Nov 02 20:11:48.880676 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:48.880235 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" event={"ID":"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab","Type":"ContainerStarted","Data":"e697831be0262ca5015706fa8106f05ee08b9119a7fe864b231253fc883d14cc"} Nov 02 20:11:49.570081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:49.570052 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:49.571438 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:49.571414 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:11:49.571529 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:49.571460 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 20:11:50.569512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:50.569480 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:11:50.569946 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:50.569533 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 20:11:51.569323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:51.569258 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:11:51.569493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:51.569351 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 20:11:52.570102 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:52.570069 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:11:52.570507 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:52.570122 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 20:11:53.569623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:53.569588 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:11:53.569819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:53.569651 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 20:11:54.570177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:54.570139 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 20:11:54.570617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:54.570192 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 20:11:55.573560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:55.573528 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:55.574030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:55.574014 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:55.574689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:55.574669 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:11:55.684394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:11:55.684327 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" podStartSLOduration=8.684312833 podStartE2EDuration="8.684312833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:49.049616243 +0000 UTC m=+1151.660268334" watchObservedRunningTime="2026-11-02 20:11:55.684312833 +0000 UTC m=+1158.294964924" Nov 02 20:12:01.396316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.396266 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2"] Nov 02 20:12:01.400228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.400208 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.443060 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.443029 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f"] Nov 02 20:12:01.449219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.449188 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.495495 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.495461 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2"] Nov 02 20:12:01.501751 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.501718 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.501899 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.501760 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.501899 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.501786 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.501973 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.501943 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9r4f\" (UniqueName: \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-kube-api-access-d9r4f\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.502058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502040 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.502146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502073 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.502146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502115 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.502249 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502232 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.502334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502306 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.502441 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502420 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1d3714-9b66-41db-be48-ec8495162ff3-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.502502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502465 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.502568 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502551 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6h5g\" (UniqueName: \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-kube-api-access-h6h5g\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.502625 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502607 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.502684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502670 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.502736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502715 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.502819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502804 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.502876 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502836 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eae2e1-db77-4773-89f6-2ab7cee55116-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.502940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.502924 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.543038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.542998 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f"] Nov 02 20:12:01.603906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.603873 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.604094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.603929 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.604094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.603969 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.604094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.603992 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eae2e1-db77-4773-89f6-2ab7cee55116-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.604094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604039 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.604094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604074 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.604415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604103 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.604415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604129 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.604415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604177 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9r4f\" (UniqueName: \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-kube-api-access-d9r4f\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.604415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604242 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.604415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604268 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.604415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604331 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.604415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604383 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.604752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604425 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.604752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604490 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1d3714-9b66-41db-be48-ec8495162ff3-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.604752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604524 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.604752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604576 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6h5g\" (UniqueName: \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-kube-api-access-h6h5g\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.604752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.604604 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.605261 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.605232 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.605836 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.605811 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.606173 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.606144 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.606488 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.606461 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.606583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.606497 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.606709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.606468 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1d3714-9b66-41db-be48-ec8495162ff3-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.606883 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.606858 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-credential-socket\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.606993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.606969 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-certs\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.606993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.606984 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eae2e1-db77-4773-89f6-2ab7cee55116-istiod-ca-cert\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.607223 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.607205 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-data\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.610428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.610405 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.611144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.611117 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.611239 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.611150 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-envoy\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.611760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.611721 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-podinfo\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.658675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.658611 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9r4f\" (UniqueName: \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-kube-api-access-d9r4f\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.658675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.658646 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.658936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.658919 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-token\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.659455 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.659436 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6h5g\" (UniqueName: \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-kube-api-access-h6h5g\") pod \"e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:01.714845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.714812 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:01.765537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:01.765501 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:02.129600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.129566 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2"] Nov 02 20:12:02.130349 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:12:02.130320 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1d3714_9b66_41db_be48_ec8495162ff3.slice/crio-6b7f896e43dbc985d93b6a135d6a155f30986ea8fd17d14be83edeb00405716d WatchSource:0}: Error finding container 6b7f896e43dbc985d93b6a135d6a155f30986ea8fd17d14be83edeb00405716d: Status 404 returned error can't find the container with id 6b7f896e43dbc985d93b6a135d6a155f30986ea8fd17d14be83edeb00405716d Nov 02 20:12:02.132732 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.132699 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:02.132799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.132767 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:02.132841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.132799 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:02.180427 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.180399 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f"] Nov 02 20:12:02.180889 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:12:02.180860 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64eae2e1_db77_4773_89f6_2ab7cee55116.slice/crio-f3a0f69587e8eac66309099994a05d624efba7cb196d7166d96ef011aff2dfc4 WatchSource:0}: Error finding container f3a0f69587e8eac66309099994a05d624efba7cb196d7166d96ef011aff2dfc4: Status 404 returned error can't find the container with id f3a0f69587e8eac66309099994a05d624efba7cb196d7166d96ef011aff2dfc4 Nov 02 20:12:02.183477 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.183445 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:02.183578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.183525 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:02.183578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.183571 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:02.983295 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.982843 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" event={"ID":"64eae2e1-db77-4773-89f6-2ab7cee55116","Type":"ContainerStarted","Data":"f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e"} Nov 02 20:12:02.983295 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.982921 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" event={"ID":"64eae2e1-db77-4773-89f6-2ab7cee55116","Type":"ContainerStarted","Data":"f3a0f69587e8eac66309099994a05d624efba7cb196d7166d96ef011aff2dfc4"} Nov 02 20:12:02.984867 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.984840 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" event={"ID":"7c1d3714-9b66-41db-be48-ec8495162ff3","Type":"ContainerStarted","Data":"a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913"} Nov 02 20:12:02.984969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:02.984890 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" event={"ID":"7c1d3714-9b66-41db-be48-ec8495162ff3","Type":"ContainerStarted","Data":"6b7f896e43dbc985d93b6a135d6a155f30986ea8fd17d14be83edeb00405716d"} Nov 02 20:12:03.195916 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:03.195834 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" podStartSLOduration=2.195820176 podStartE2EDuration="2.195820176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:03.147886997 +0000 UTC m=+1165.758539089" watchObservedRunningTime="2026-11-02 20:12:03.195820176 +0000 UTC m=+1165.806472266" Nov 02 20:12:03.283073 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:03.282950 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" podStartSLOduration=2.28293361 podStartE2EDuration="2.28293361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:03.274810846 +0000 UTC m=+1165.885462938" watchObservedRunningTime="2026-11-02 20:12:03.28293361 +0000 UTC m=+1165.893585700" Nov 02 20:12:03.715083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:03.715052 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:03.716528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:03.716499 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:12:03.716667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:03.716549 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:12:03.771297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:03.770827 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:03.771297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:03.770979 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:12:03.771297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:03.771028 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:12:04.715917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:04.715873 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:12:04.716373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:04.715934 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:12:04.773298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:04.773247 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:12:04.773566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:04.773537 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:12:05.715426 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:05.715394 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:12:05.715599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:05.715446 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:12:05.775926 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:05.775868 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:12:05.776401 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:05.775951 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:12:06.715759 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:06.715719 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:12:06.715942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:06.715793 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:12:06.767702 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:06.767662 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:12:06.768028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:06.767993 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:12:07.715628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:07.715596 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:12:07.716012 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:07.715645 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:12:07.766785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:07.766745 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:12:07.767065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:07.767033 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:12:08.715360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:08.715324 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:12:08.715536 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:08.715390 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:12:08.766138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:08.766091 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:12:08.766547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:08.766162 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:12:09.720008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:09.719978 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:09.720420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:09.720398 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:09.721022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:09.721003 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:12:09.770743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:09.770718 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:09.771148 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:09.770988 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:09.771774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:09.771757 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:12:13.213284 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.213247 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68"] Nov 02 20:12:13.217527 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.217505 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.295090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.288111 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68"] Nov 02 20:12:13.295374 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.295303 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-dshm\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.295457 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.295352 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-tmp-dir\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.295526 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.295480 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x64th\" (UniqueName: \"kubernetes.io/projected/fa70b62b-4591-401c-bb78-5dfdde976042-kube-api-access-x64th\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.295585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.295532 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-home\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.295658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.295636 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa70b62b-4591-401c-bb78-5dfdde976042-tls-certs\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.295906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.295885 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-model-cache\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.312960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.312935 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"openshift-service-ca.crt\"" Nov 02 20:12:13.313160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.313144 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"kube-root-ca.crt\"" Nov 02 20:12:13.337650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.337625 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"default-dockercfg-gpd7p\"" Nov 02 20:12:13.337823 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.337756 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9fcedd\"/\"distinct-2-w2-9fcedd-kserve-self-signed-certs\"" Nov 02 20:12:13.408971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.407874 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64th\" (UniqueName: \"kubernetes.io/projected/fa70b62b-4591-401c-bb78-5dfdde976042-kube-api-access-x64th\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.408971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.407933 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-home\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.408971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.407996 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa70b62b-4591-401c-bb78-5dfdde976042-tls-certs\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.408971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.408040 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-model-cache\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.408971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.408147 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-dshm\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.408971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.408184 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-tmp-dir\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.408971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.408661 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-tmp-dir\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.409582 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.409536 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-home\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.410101 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.410078 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-model-cache\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.412460 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.412435 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-dshm\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.412815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.412792 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa70b62b-4591-401c-bb78-5dfdde976042-tls-certs\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.463910 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.463792 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64th\" (UniqueName: \"kubernetes.io/projected/fa70b62b-4591-401c-bb78-5dfdde976042-kube-api-access-x64th\") pod \"distinct-2-w2-9fcedd-kserve-557948485f-4zf68\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.530992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.530956 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:13.973031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:13.973001 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68"] Nov 02 20:12:13.973748 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:12:13.973721 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa70b62b_4591_401c_bb78_5dfdde976042.slice/crio-44b47bacd7fee0c74c9a77ff4dbc35a1cd0ad6ead15233f406e8fe3169c42026 WatchSource:0}: Error finding container 44b47bacd7fee0c74c9a77ff4dbc35a1cd0ad6ead15233f406e8fe3169c42026: Status 404 returned error can't find the container with id 44b47bacd7fee0c74c9a77ff4dbc35a1cd0ad6ead15233f406e8fe3169c42026 Nov 02 20:12:14.061050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:14.061016 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" event={"ID":"fa70b62b-4591-401c-bb78-5dfdde976042","Type":"ContainerStarted","Data":"38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8"} Nov 02 20:12:14.061050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:14.061059 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" event={"ID":"fa70b62b-4591-401c-bb78-5dfdde976042","Type":"ContainerStarted","Data":"44b47bacd7fee0c74c9a77ff4dbc35a1cd0ad6ead15233f406e8fe3169c42026"} Nov 02 20:12:14.168402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:14.168350 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podStartSLOduration=1.168336337 podStartE2EDuration="1.168336337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:14.166131159 +0000 UTC m=+1176.776783250" watchObservedRunningTime="2026-11-02 20:12:14.168336337 +0000 UTC m=+1176.778988427" Nov 02 20:12:14.531684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:14.531653 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:18.238787 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:12:18.238705 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:12:23.532489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:23.532458 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:23.545355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:23.545336 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:24.134161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:24.134135 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:12:30.294714 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:30.294678 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:12:46.106703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.106668 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c"] Nov 02 20:12:46.111945 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.111926 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.196018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.195984 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.196018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.196021 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.196269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.196043 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.196269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.196064 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.196269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.196185 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxml\" (UniqueName: \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-kube-api-access-ssxml\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.196504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.196333 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.196504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.196400 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.196504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.196430 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.196504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.196485 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.198691 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.198649 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9fcedd-openshift-default-dockercfg-rmhls\"" Nov 02 20:12:46.211307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.211251 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c"] Nov 02 20:12:46.297492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.297456 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.297492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.297496 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.297758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.297522 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.297758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.297662 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.297758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.297691 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.297758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.297739 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.297963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.297771 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.297963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.297814 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxml\" (UniqueName: \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-kube-api-access-ssxml\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.297963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.297925 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.297963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.297955 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.298555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.298531 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.298820 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.298794 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.298820 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.298810 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.298939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.298864 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.300420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.300397 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.301424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.301404 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.352831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.352798 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxml\" (UniqueName: \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-kube-api-access-ssxml\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.354548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.354379 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.359804 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.359745 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4"] Nov 02 20:12:46.365979 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.365962 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.399303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.399176 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-tmp-dir\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.399549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.399527 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-home\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.399665 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.399643 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-dshm\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.399748 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.399717 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead97984-d504-497d-b1ca-c2781eace145-tls-certs\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.399840 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.399825 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpsb2\" (UniqueName: \"kubernetes.io/projected/ead97984-d504-497d-b1ca-c2781eace145-kube-api-access-tpsb2\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.399959 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.399945 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-model-cache\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.416649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.416616 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4"] Nov 02 20:12:46.416969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.416949 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-035e47\"/\"distinct-2-w0-035e47-kserve-self-signed-certs\"" Nov 02 20:12:46.425978 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.425950 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:46.503525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.501351 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-tmp-dir\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.503525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.501408 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-home\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.503525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.501448 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-dshm\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.503525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.501505 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead97984-d504-497d-b1ca-c2781eace145-tls-certs\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.503525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.501588 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpsb2\" (UniqueName: \"kubernetes.io/projected/ead97984-d504-497d-b1ca-c2781eace145-kube-api-access-tpsb2\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.503525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.501705 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-model-cache\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.503525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.502191 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-model-cache\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.503525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.502482 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-tmp-dir\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.503525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.502723 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-home\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.511310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.509574 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-dshm\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.511310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.509617 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead97984-d504-497d-b1ca-c2781eace145-tls-certs\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.557671 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.557641 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpsb2\" (UniqueName: \"kubernetes.io/projected/ead97984-d504-497d-b1ca-c2781eace145-kube-api-access-tpsb2\") pod \"distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.687404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.687324 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:46.848189 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.848155 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c"] Nov 02 20:12:46.849551 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:12:46.849525 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b51926_7345_4f60_b2b7_0a82fbcca4c5.slice/crio-0d90e6b1cc1e665385dbaa14f186a025b6607b1a685970c7a491293c3b6a768e WatchSource:0}: Error finding container 0d90e6b1cc1e665385dbaa14f186a025b6607b1a685970c7a491293c3b6a768e: Status 404 returned error can't find the container with id 0d90e6b1cc1e665385dbaa14f186a025b6607b1a685970c7a491293c3b6a768e Nov 02 20:12:46.851969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.851939 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:46.852050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.852006 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:46.852050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:46.852041 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:47.090531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.090501 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4"] Nov 02 20:12:47.090700 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:12:47.090679 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podead97984_d504_497d_b1ca_c2781eace145.slice/crio-f8b392b8d81957069be6bffcf42803d3338bc4b766388e4f3174c660fe88824a WatchSource:0}: Error finding container f8b392b8d81957069be6bffcf42803d3338bc4b766388e4f3174c660fe88824a: Status 404 returned error can't find the container with id f8b392b8d81957069be6bffcf42803d3338bc4b766388e4f3174c660fe88824a Nov 02 20:12:47.282141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.282100 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" event={"ID":"88b51926-7345-4f60-b2b7-0a82fbcca4c5","Type":"ContainerStarted","Data":"09214e2d9acfea55be9fdfaf031a85b57db48b51307fcfe75f1c6b94ccd17726"} Nov 02 20:12:47.282648 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.282170 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" event={"ID":"88b51926-7345-4f60-b2b7-0a82fbcca4c5","Type":"ContainerStarted","Data":"0d90e6b1cc1e665385dbaa14f186a025b6607b1a685970c7a491293c3b6a768e"} Nov 02 20:12:47.284171 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.284134 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" event={"ID":"ead97984-d504-497d-b1ca-c2781eace145","Type":"ContainerStarted","Data":"8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a"} Nov 02 20:12:47.284318 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.284181 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" event={"ID":"ead97984-d504-497d-b1ca-c2781eace145","Type":"ContainerStarted","Data":"f8b392b8d81957069be6bffcf42803d3338bc4b766388e4f3174c660fe88824a"} Nov 02 20:12:47.419371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.416950 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" podStartSLOduration=1.4169290669999999 podStartE2EDuration="1.416929067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:47.390199202 +0000 UTC m=+1210.000851293" watchObservedRunningTime="2026-11-02 20:12:47.416929067 +0000 UTC m=+1210.027581160" Nov 02 20:12:47.439301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.435743 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:47.439301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.435887 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:12:47.439301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.435931 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:12:47.521578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.521523 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podStartSLOduration=1.521500702 podStartE2EDuration="1.521500702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:47.520549326 +0000 UTC m=+1210.131201418" watchObservedRunningTime="2026-11-02 20:12:47.521500702 +0000 UTC m=+1210.132152794" Nov 02 20:12:47.688614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:47.688523 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:48.264899 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:12:48.264859 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:12:48.438316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:48.432688 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:12:48.438316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:48.432753 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:12:49.437301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:49.426338 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:12:49.437301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:49.426416 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:12:50.449300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:50.446623 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:12:50.449300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:50.446685 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:12:51.438690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:51.438654 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:12:51.438975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:51.438948 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:12:52.449301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:52.448710 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:12:52.449301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:52.448779 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:12:53.426697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:53.426662 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:12:53.426851 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:53.426715 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:12:54.431084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:54.431041 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:54.431604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:54.431322 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:54.431996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:54.431976 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:12:56.688112 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:56.688081 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:56.700940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:56.700915 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:12:57.362427 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:12:57.362393 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:13:01.558612 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:01.557579 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:16.647577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.647539 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s"] Nov 02 20:13:16.651920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.651897 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.739714 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.739678 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.739714 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.739713 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.739969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.739839 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.739969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.739916 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.739969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.739966 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.740139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.740036 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfbw\" (UniqueName: \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-kube-api-access-xnfbw\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.740139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.740074 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.740139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.740103 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.740139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.740129 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841126 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841090 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841155 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841223 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfbw\" (UniqueName: \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-kube-api-access-xnfbw\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841251 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841306 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841340 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841444 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841471 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841525 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841651 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.841923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.841892 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.842199 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.842174 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.842539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.842513 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.842894 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.842848 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.844620 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.844595 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.844789 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.844771 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.845141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.845114 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s"] Nov 02 20:13:16.894831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.894797 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.895129 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.895105 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfbw\" (UniqueName: \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-kube-api-access-xnfbw\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:16.965832 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:16.965751 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:17.277925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.277848 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj"] Nov 02 20:13:17.283183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.283160 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.337427 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.337393 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj"] Nov 02 20:13:17.352100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.352064 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.352286 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.352173 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.352286 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.352255 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.352413 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.352321 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.352468 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.352428 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkv9r\" (UniqueName: \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-kube-api-access-lkv9r\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.352546 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.352528 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.352613 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.352564 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.352613 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.352603 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.352712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.352655 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.427857 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.427825 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s"] Nov 02 20:13:17.429338 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:13:17.429309 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6ae9d0_3900_4e9b_ac3b_a1ccb9f1da22.slice/crio-b2ae9875fca301f42f5a3cff146838b50c6365338b41842698d7b55ae12d7953 WatchSource:0}: Error finding container b2ae9875fca301f42f5a3cff146838b50c6365338b41842698d7b55ae12d7953: Status 404 returned error can't find the container with id b2ae9875fca301f42f5a3cff146838b50c6365338b41842698d7b55ae12d7953 Nov 02 20:13:17.431755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.431727 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:17.431835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.431793 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:17.431835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.431829 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:17.453513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.453490 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.453623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.453525 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.453623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.453555 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.453623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.453596 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.453780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.453654 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkv9r\" (UniqueName: \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-kube-api-access-lkv9r\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.453780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.453737 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.453780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.453771 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.453944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.453809 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.453944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.453856 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.453944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.453940 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.454218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.454196 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.454336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.454253 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.454473 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.454454 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.454473 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.454458 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.455994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.455973 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.456077 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.456054 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.506970 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.506931 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.507141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.507125 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkv9r\" (UniqueName: \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-kube-api-access-lkv9r\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.587514 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.587476 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" event={"ID":"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22","Type":"ContainerStarted","Data":"7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8"} Nov 02 20:13:17.587705 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.587527 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" event={"ID":"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22","Type":"ContainerStarted","Data":"b2ae9875fca301f42f5a3cff146838b50c6365338b41842698d7b55ae12d7953"} Nov 02 20:13:17.600736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.600711 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:17.702326 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.702218 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" podStartSLOduration=1.7021995890000001 podStartE2EDuration="1.702199589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:17.696337329 +0000 UTC m=+1240.306989440" watchObservedRunningTime="2026-11-02 20:13:17.702199589 +0000 UTC m=+1240.312851682" Nov 02 20:13:17.913578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.913484 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv"] Nov 02 20:13:17.919643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.919620 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:17.961146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.961049 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdgvc\" (UniqueName: \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-kube-api-access-mdgvc\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:17.961146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.961117 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:17.961469 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.961175 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27f161f-eb87-4410-a556-400a65eab338-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:17.961469 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.961237 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:17.961469 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.961349 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:17.961469 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.961390 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27f161f-eb87-4410-a556-400a65eab338-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:17.961469 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.961424 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:17.961469 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.961468 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:17.961786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.961526 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:17.982808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:17.978378 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:18.007509 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.007468 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv"] Nov 02 20:13:18.021851 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:13:18.021814 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1f59bf_d4de_458d_ba74_1aaeb2744888.slice/crio-b40bed3a5b8d745ad3a2e35e17079c73115d5b8cd279c597ad3a1c402c95494d WatchSource:0}: Error finding container b40bed3a5b8d745ad3a2e35e17079c73115d5b8cd279c597ad3a1c402c95494d: Status 404 returned error can't find the container with id b40bed3a5b8d745ad3a2e35e17079c73115d5b8cd279c597ad3a1c402c95494d Nov 02 20:13:18.022831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.022793 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj"] Nov 02 20:13:18.024849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.024806 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:18.024959 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.024893 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:18.024959 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.024939 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.068671 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.068815 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdgvc\" (UniqueName: \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-kube-api-access-mdgvc\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.068867 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.068921 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27f161f-eb87-4410-a556-400a65eab338-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.068972 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.069046 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.069084 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27f161f-eb87-4410-a556-400a65eab338-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.069119 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.069160 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.072427 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-certs\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.073344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.072684 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-data\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.074203 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.073628 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27f161f-eb87-4410-a556-400a65eab338-istiod-ca-cert\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.074203 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.073913 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.074350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.074200 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-credential-socket\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.077079 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.077049 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-envoy\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.082326 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.082262 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27f161f-eb87-4410-a556-400a65eab338-istio-podinfo\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.126005 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.125978 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdgvc\" (UniqueName: \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-kube-api-access-mdgvc\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.128321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.126323 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-istio-token\") pod \"e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.244425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.237711 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:18.260881 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.260835 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7"] Nov 02 20:13:18.272192 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.271729 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.279149 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:13:18.279066 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:13:18.289986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.289959 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7"] Nov 02 20:13:18.373498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.373197 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.373498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.373243 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.373498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.373351 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.373498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.373389 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.373498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.373453 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.374333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.373661 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47j2\" (UniqueName: \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-kube-api-access-z47j2\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.374333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.373701 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.374333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.373785 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.374333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.373840 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.475014 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.474938 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47j2\" (UniqueName: \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-kube-api-access-z47j2\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.475219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.475206 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.475778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.475602 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.476113 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.476089 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.476252 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.476158 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.476252 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.476237 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.476386 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.476267 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.476386 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.476359 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.476499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.476394 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.476499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.476465 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.476729 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.476701 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.476908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.476882 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.477037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.476970 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.477268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.477245 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.478905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.478880 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.479936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.479914 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.529368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.529296 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47j2\" (UniqueName: \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-kube-api-access-z47j2\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.529546 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.529524 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.597776 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.597747 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:18.599200 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.599158 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" event={"ID":"9b1f59bf-d4de-458d-ba74-1aaeb2744888","Type":"ContainerStarted","Data":"e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298"} Nov 02 20:13:18.599355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.599219 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" event={"ID":"9b1f59bf-d4de-458d-ba74-1aaeb2744888","Type":"ContainerStarted","Data":"b40bed3a5b8d745ad3a2e35e17079c73115d5b8cd279c597ad3a1c402c95494d"} Nov 02 20:13:18.601871 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.601670 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:18.675414 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.675378 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv"] Nov 02 20:13:18.678930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.677932 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:18.678930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.678015 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:18.678930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.678064 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:18.718870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.717789 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podStartSLOduration=1.717767871 podStartE2EDuration="1.717767871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:18.71659779 +0000 UTC m=+1241.327249882" watchObservedRunningTime="2026-11-02 20:13:18.717767871 +0000 UTC m=+1241.328419964" Nov 02 20:13:18.985301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.984788 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:13:18.985301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:18.984854 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:13:19.026222 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.026190 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7"] Nov 02 20:13:19.057983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.051240 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:19.057983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.051386 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:19.057983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.051450 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:19.603010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.602629 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:13:19.603010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.602692 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:13:19.615314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.615198 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" event={"ID":"1cf5b8ba-4340-4df2-9763-2c2f071862d5","Type":"ContainerStarted","Data":"4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98"} Nov 02 20:13:19.615314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.615265 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" event={"ID":"1cf5b8ba-4340-4df2-9763-2c2f071862d5","Type":"ContainerStarted","Data":"f82c78f85bc1ce06d7ea92c5d074d3cf8efe35f14d406188aa131ed98eb0225f"} Nov 02 20:13:19.617268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.617236 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" event={"ID":"d27f161f-eb87-4410-a556-400a65eab338","Type":"ContainerStarted","Data":"92f847533c7ce9efa56b3d612da0c7201bbd3bb3fdf9ff17c1a6545c71a08afb"} Nov 02 20:13:19.617405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.617300 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" event={"ID":"d27f161f-eb87-4410-a556-400a65eab338","Type":"ContainerStarted","Data":"9d505d928587649cffcacff674f161911478d2dfc63eada8c1a36f7320ae5394"} Nov 02 20:13:19.733102 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.733017 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" podStartSLOduration=1.732997252 podStartE2EDuration="1.732997252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:19.72704016 +0000 UTC m=+1242.337692252" watchObservedRunningTime="2026-11-02 20:13:19.732997252 +0000 UTC m=+1242.343649344" Nov 02 20:13:19.875354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.869794 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" podStartSLOduration=2.86975825 podStartE2EDuration="2.86975825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:19.86088553 +0000 UTC m=+1242.471537623" watchObservedRunningTime="2026-11-02 20:13:19.86975825 +0000 UTC m=+1242.480410340" Nov 02 20:13:19.969742 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.969699 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:13:19.984110 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:19.970044 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:13:20.251491 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:20.251344 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:20.279303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:20.256404 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:13:20.279303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:20.256466 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:13:20.604334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:20.601926 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:20.604334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:20.602049 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:13:20.604334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:20.602092 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:13:20.604334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:20.603710 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:13:20.604334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:20.603754 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:13:20.976908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:20.976799 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:13:20.977560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:20.977508 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:13:21.141038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.140986 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.202415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.202369 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.264368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.248667 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:13:21.269308 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.268411 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:13:21.348527 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.348481 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.600403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.599697 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:13:21.600403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.599761 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:13:21.604954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.604911 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:13:21.625940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.625901 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:13:21.626225 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.626190 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:13:21.708437 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.708386 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:21.956467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.956364 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 20:13:21.988182 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.988145 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:13:21.988832 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:21.988799 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:13:22.043949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.043903 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.147077 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.147027 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:13:22.165476 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.165439 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:13:22.262361 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.254865 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:13:22.279302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.275361 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:13:22.358173 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.358124 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 02 20:13:22.482306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.478743 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:13:22.579398 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.569723 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.675441 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.675394 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:13:22.675740 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.675704 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:13:22.675939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.675903 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:13:22.676431 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.676414 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:13:22.681361 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.675041 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:13:22.682626 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.674908 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:13:22.682791 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.682768 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:13:22.746317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.722383 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:13:22.752934 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.752905 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:13:22.986529 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.986412 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:22.989374 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:22.989343 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:13:23.057301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.020553 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:13:23.151333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.100137 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:13:23.151333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.058405 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.151333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.105735 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:13:23.170157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.170100 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.180842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.180792 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.251356 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.247746 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:13:23.289447 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.289400 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:13:23.511352 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.510353 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.615301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.614827 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.653789 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.615579 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 02 20:13:23.653789 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.630220 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:13:23.653789 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.630309 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:13:23.653789 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.631071 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:13:23.653789 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.631110 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:13:23.680550 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.679158 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.698419 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.698368 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:23.762363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.762233 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:13:23.824036 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.823998 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:13:24.093368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.947430 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:13:24.093368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:23.998780 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.142407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.142345 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.164302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.146496 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:13:24.164302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.160357 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:13:24.196300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.189843 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.214663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.197175 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.240308 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.215242 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:13:24.241330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.198528 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.665418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.349537 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:13:24.665745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.665709 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:13:24.665869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.566508 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.665972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.624900 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:13:24.673308 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.511437 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.677337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.648424 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:13:24.677337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.677246 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:13:24.677337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.654637 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:13:24.677337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.677328 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:13:24.706104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.698379 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.706104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.698537 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.766313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.765215 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:24.967114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.967014 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:13:24.967466 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:24.967432 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:13:25.002773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.001363 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.046688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.045493 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:13:25.146779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.146728 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 20:13:25.216672 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.216337 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.220795 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.217813 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:13:25.329673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.329436 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:13:25.336411 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.336359 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.336708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.336660 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:13:25.515818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.515719 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:13:25.615311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.614500 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:13:25.615311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.614937 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.615311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.614990 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:13:25.615311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.615183 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:13:25.615311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.615206 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:13:25.619347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.619323 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:13:25.620905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.620881 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:13:25.705035 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.704976 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:25.708052 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.708015 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:13:25.785249 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.785154 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:13:25.924869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:25.923094 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:13:26.018600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:26.018563 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:13:26.249546 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:26.249512 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:13:26.250157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:26.250108 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:13:26.473825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:26.473795 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:26.474375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:26.474348 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:26.476259 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:26.476242 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:13:26.598522 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:26.598490 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:13:26.598683 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:26.598552 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:13:26.601564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:26.601535 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:13:26.601679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:26.601591 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:13:27.245649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:27.245610 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:13:27.245945 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:27.245904 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:13:27.609387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:27.609353 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:27.611132 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:27.611104 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:27.616030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:27.616005 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:13:27.698065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:27.698031 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:27.699343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:27.699322 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:27.706959 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:27.706933 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:13:28.249915 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:28.249886 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:28.250174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:28.250152 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:28.250863 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:28.250844 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:13:31.776435 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.776398 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq"] Nov 02 20:13:31.780868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.780844 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.861186 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.861149 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.861350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.861194 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2853491-636a-4980-9fe1-a90a3a06a5f0-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.861350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.861219 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.861460 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.861375 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.861460 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.861420 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.861582 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.861558 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m4lt\" (UniqueName: \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-kube-api-access-4m4lt\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.861642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.861602 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.861700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.861687 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.861746 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.861718 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.879713 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.879687 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq"] Nov 02 20:13:31.963287 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963252 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.963479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963325 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2853491-636a-4980-9fe1-a90a3a06a5f0-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.963479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963352 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.963479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963407 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.963479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963451 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.963690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963531 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m4lt\" (UniqueName: \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-kube-api-access-4m4lt\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.963690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963560 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.963690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963591 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.963690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963615 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.963690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963650 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.964008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.963983 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-credential-socket\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.964099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.964074 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-data\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.964153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.964102 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2853491-636a-4980-9fe1-a90a3a06a5f0-istiod-ca-cert\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.964215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.964151 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-certs\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.966285 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.966251 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-envoy\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:31.966520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:31.966501 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-podinfo\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:32.016771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:32.016742 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m4lt\" (UniqueName: \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-kube-api-access-4m4lt\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:32.016919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:32.016745 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-token\") pod \"e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:32.094382 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:32.094349 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:32.516924 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:32.516893 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq"] Nov 02 20:13:32.517090 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:13:32.517065 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2853491_636a_4980_9fe1_a90a3a06a5f0.slice/crio-54f3fa91e3226dd63015cb0c94a14e4b537f15cb46a20f6233f9f39654219a30 WatchSource:0}: Error finding container 54f3fa91e3226dd63015cb0c94a14e4b537f15cb46a20f6233f9f39654219a30: Status 404 returned error can't find the container with id 54f3fa91e3226dd63015cb0c94a14e4b537f15cb46a20f6233f9f39654219a30 Nov 02 20:13:32.519611 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:32.519580 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:32.519689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:32.519643 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:32.519689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:32.519675 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:32.693602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:32.693556 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" event={"ID":"d2853491-636a-4980-9fe1-a90a3a06a5f0","Type":"ContainerStarted","Data":"69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9"} Nov 02 20:13:32.693893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:32.693639 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" event={"ID":"d2853491-636a-4980-9fe1-a90a3a06a5f0","Type":"ContainerStarted","Data":"54f3fa91e3226dd63015cb0c94a14e4b537f15cb46a20f6233f9f39654219a30"} Nov 02 20:13:32.806167 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:32.806119 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" podStartSLOduration=1.806104119 podStartE2EDuration="1.806104119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:32.804425417 +0000 UTC m=+1255.415077509" watchObservedRunningTime="2026-11-02 20:13:32.806104119 +0000 UTC m=+1255.416756210" Nov 02 20:13:33.094891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.094810 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:33.096448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.096421 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:33.096564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.096466 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:33.361987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.361893 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp"] Nov 02 20:13:33.367128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.367057 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.415182 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.415137 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp"] Nov 02 20:13:33.496727 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.496672 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plzfd\" (UniqueName: \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-kube-api-access-plzfd\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.496932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.496744 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.496932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.496867 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.496932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.496904 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.497117 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.496990 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.497117 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.497054 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.497117 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.497110 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.497292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.497135 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.497355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.497299 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.598776 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.598742 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.598971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.598882 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzfd\" (UniqueName: \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-kube-api-access-plzfd\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.598971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.598917 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.599126 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.599104 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.599181 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.599153 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.599237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.599192 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.599237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.599223 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.599404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.599367 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.599600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.599574 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.599890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.599571 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-certs\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.600026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.599391 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-credential-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.600107 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.599643 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-socket\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.600107 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.599892 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-data\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.600213 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.600186 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istiod-ca-cert\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.601584 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.601551 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-envoy\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.601715 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.601694 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-podinfo\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.654551 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.654475 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzfd\" (UniqueName: \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-kube-api-access-plzfd\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.654908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.654883 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-token\") pod \"e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:33.681741 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:33.681702 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:34.095318 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:34.095269 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:34.095727 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:34.095338 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:34.157165 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:34.157119 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp"] Nov 02 20:13:34.157714 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:13:34.157687 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3c6aae_d44a_4a78_adb7_48a19e857afb.slice/crio-751a16996fc7f8232467dae8dacf4c981d01273d4f5c784b62ff60c41635539d WatchSource:0}: Error finding container 751a16996fc7f8232467dae8dacf4c981d01273d4f5c784b62ff60c41635539d: Status 404 returned error can't find the container with id 751a16996fc7f8232467dae8dacf4c981d01273d4f5c784b62ff60c41635539d Nov 02 20:13:34.160677 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:34.160633 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:34.160777 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:34.160719 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:34.160777 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:34.160765 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:34.713431 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:34.711971 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" event={"ID":"ef3c6aae-d44a-4a78-adb7-48a19e857afb","Type":"ContainerStarted","Data":"362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9"} Nov 02 20:13:34.713431 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:34.712041 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" event={"ID":"ef3c6aae-d44a-4a78-adb7-48a19e857afb","Type":"ContainerStarted","Data":"751a16996fc7f8232467dae8dacf4c981d01273d4f5c784b62ff60c41635539d"} Nov 02 20:13:34.824115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:34.824057 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" podStartSLOduration=1.824041566 podStartE2EDuration="1.824041566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:34.822216778 +0000 UTC m=+1257.432868869" watchObservedRunningTime="2026-11-02 20:13:34.824041566 +0000 UTC m=+1257.434693657" Nov 02 20:13:35.095076 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:35.095045 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:35.095264 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:35.095094 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:35.682324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:35.682173 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:35.684011 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:35.683969 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:13:35.684132 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:35.684027 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:13:36.095586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:36.095554 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:36.095760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:36.095604 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:36.682142 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:36.682104 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:13:36.682424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:36.682388 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:13:37.095019 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:37.094989 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:37.095301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:37.095260 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:37.685896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:37.685575 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:13:37.685896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:37.685662 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:13:38.094762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:38.094725 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:38.094954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:38.094783 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:38.682838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:38.682748 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:13:38.682838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:38.682811 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:13:39.095609 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:39.095572 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:13:39.096134 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:39.095631 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:13:39.682852 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:39.682816 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:13:39.683064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:39.682868 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:13:39.885451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:39.885414 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:13:40.386424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:40.386388 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:40.386894 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:40.386740 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:40.387731 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:40.387708 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:13:40.684398 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:40.684101 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:13:40.684960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:40.684926 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:13:42.710310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:42.709628 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:42.749301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:42.712363 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:13:42.749301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:42.712433 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:42.903303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:42.852669 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:42.903303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:42.854531 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:42.903303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:42.858569 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:13:48.246756 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:13:48.246667 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:13:51.048951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:51.048909 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r"] Nov 02 20:13:51.095008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:51.094970 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-429d3fa5-zr28r"] Nov 02 20:13:51.508658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:13:51.508619 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37f4181-9977-4ac5-a86e-3ae635912d09" path="/var/lib/kubelet/pods/a37f4181-9977-4ac5-a86e-3ae635912d09/volumes" Nov 02 20:14:02.615132 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.615083 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w"] Nov 02 20:14:02.619254 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.619229 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.688796 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.688762 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.688965 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.688808 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.688965 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.688949 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d1fe52-533d-4ace-b8bd-defa734d3278-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.689052 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.688998 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.689091 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.689071 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9k57\" (UniqueName: \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-kube-api-access-h9k57\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.689149 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.689094 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.689253 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.689230 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.689351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.689333 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.689410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.689400 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.713368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.713332 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w"] Nov 02 20:14:02.790485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.790448 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9k57\" (UniqueName: \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-kube-api-access-h9k57\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.790485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.790489 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.790690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.790544 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.790690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.790571 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.790690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.790639 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.790690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.790668 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.790893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.790710 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.790893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.790820 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d1fe52-533d-4ace-b8bd-defa734d3278-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.790893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.790877 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.791071 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.790971 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-data\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.791071 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.791025 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.791181 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.791109 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-credential-socket\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.791259 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.791241 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-certs\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.791612 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.791588 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d1fe52-533d-4ace-b8bd-defa734d3278-istiod-ca-cert\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.792989 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.792961 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-envoy\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.793184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.793167 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-podinfo\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.844057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.844017 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-token\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.844199 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.844072 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9k57\" (UniqueName: \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-kube-api-access-h9k57\") pod \"e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:02.936111 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:02.936022 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:03.355802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:03.355772 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w"] Nov 02 20:14:03.359679 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:14:03.359650 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d1fe52_533d_4ace_b8bd_defa734d3278.slice/crio-bd7db870f665c0f26152f73e2afa13b7db23d2c5d0fc68e6cca9f7444dd724d4 WatchSource:0}: Error finding container bd7db870f665c0f26152f73e2afa13b7db23d2c5d0fc68e6cca9f7444dd724d4: Status 404 returned error can't find the container with id bd7db870f665c0f26152f73e2afa13b7db23d2c5d0fc68e6cca9f7444dd724d4 Nov 02 20:14:03.361740 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:03.361707 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:03.361824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:03.361769 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:03.361824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:03.361799 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:04.027627 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:04.027591 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" event={"ID":"03d1fe52-533d-4ace-b8bd-defa734d3278","Type":"ContainerStarted","Data":"cbc248d79450376a5ea402f5444c052d228f6148f1951431954125a176ae2002"} Nov 02 20:14:04.028105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:04.027651 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" event={"ID":"03d1fe52-533d-4ace-b8bd-defa734d3278","Type":"ContainerStarted","Data":"bd7db870f665c0f26152f73e2afa13b7db23d2c5d0fc68e6cca9f7444dd724d4"} Nov 02 20:14:04.184183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:04.184123 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podStartSLOduration=2.184110058 podStartE2EDuration="2.184110058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:04.182863217 +0000 UTC m=+1286.793515305" watchObservedRunningTime="2026-11-02 20:14:04.184110058 +0000 UTC m=+1286.794762149" Nov 02 20:14:04.937770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:04.937729 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:04.951533 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:04.951493 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:14:04.951751 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:04.951575 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 20:14:05.951780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:05.951743 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:14:05.952190 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:05.951830 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 20:14:06.951303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:06.941386 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:14:06.951303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:06.941449 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 20:14:07.054198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:07.054159 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq"] Nov 02 20:14:07.091557 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:07.091521 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78373750-jlbpq"] Nov 02 20:14:07.509268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:07.509233 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a011e758-72da-4b69-825d-3d5998bfd021" path="/var/lib/kubelet/pods/a011e758-72da-4b69-825d-3d5998bfd021/volumes" Nov 02 20:14:07.944426 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:07.944382 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:14:07.944736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:07.944703 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 20:14:08.153302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.150818 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9"] Nov 02 20:14:08.162143 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.162113 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.182551 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.182527 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9"] Nov 02 20:14:08.189525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.189496 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwl2\" (UniqueName: \"kubernetes.io/projected/ddb09c2e-faa8-4231-afcb-6b3e08998ead-kube-api-access-tfwl2\") pod \"maas-api-revoke-sub-955bcc67-vx5x9\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.189724 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.189706 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955bcc67-vx5x9\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.189888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.189874 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955bcc67-vx5x9\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.224650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.224565 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:14:08.224868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.224574 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:14:08.291488 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.291450 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955bcc67-vx5x9\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.291682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.291599 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955bcc67-vx5x9\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.291755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.291701 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwl2\" (UniqueName: \"kubernetes.io/projected/ddb09c2e-faa8-4231-afcb-6b3e08998ead-kube-api-access-tfwl2\") pod \"maas-api-revoke-sub-955bcc67-vx5x9\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.292122 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.292088 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955bcc67-vx5x9\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.293997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.293975 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955bcc67-vx5x9\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.353698 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.353668 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwl2\" (UniqueName: \"kubernetes.io/projected/ddb09c2e-faa8-4231-afcb-6b3e08998ead-kube-api-access-tfwl2\") pod \"maas-api-revoke-sub-955bcc67-vx5x9\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.483113 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.480525 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:08.998184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.988882 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9"] Nov 02 20:14:08.999531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.999389 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:14:08.999531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:08.999463 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 20:14:09.106077 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:09.105982 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" event={"ID":"ddb09c2e-faa8-4231-afcb-6b3e08998ead","Type":"ContainerStarted","Data":"18f4cbc2856e6dd7ec8b99064fad81694992121e57ba61ac095a4741adea4339"} Nov 02 20:14:09.943302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:09.939652 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:14:09.943302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:09.939713 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 20:14:10.180518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.180465 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" event={"ID":"ddb09c2e-faa8-4231-afcb-6b3e08998ead","Type":"ContainerStarted","Data":"0f516dcbf02159c99b586d40db96caa99f3ff1128c0eb1ff3200459117d8cca0"} Nov 02 20:14:10.298828 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.298747 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" podStartSLOduration=2.298727751 podStartE2EDuration="2.298727751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:10.294677278 +0000 UTC m=+1292.905329370" watchObservedRunningTime="2026-11-02 20:14:10.298727751 +0000 UTC m=+1292.909379842" Nov 02 20:14:10.526364 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.518157 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 02 20:14:10.614687 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.614353 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 02 20:14:10.627161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.626356 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:10.664951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.664456 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:10.682418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.682360 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:10.707449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.707408 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:14:10.937710 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.937670 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 20:14:10.937987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.937956 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 20:14:10.938900 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:10.938556 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:11.010245 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.009811 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 20:14:11.156845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.156786 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:11.222934 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.220819 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:14:11.235332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.226027 5399 generic.go:356] "Generic (PLEG): container finished" podID="ddb09c2e-faa8-4231-afcb-6b3e08998ead" containerID="0f516dcbf02159c99b586d40db96caa99f3ff1128c0eb1ff3200459117d8cca0" exitCode=0 Nov 02 20:14:11.235332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.226261 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" event={"ID":"ddb09c2e-faa8-4231-afcb-6b3e08998ead","Type":"ContainerDied","Data":"0f516dcbf02159c99b586d40db96caa99f3ff1128c0eb1ff3200459117d8cca0"} Nov 02 20:14:11.546358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.544717 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:11.626814 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.626178 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:14:11.626814 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.626765 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:11.627796 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.627157 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:14:11.718301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.717374 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:11.718301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.717783 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:14:11.721307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.718815 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:14:11.721307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.718880 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:14:11.791660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.791582 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:14:11.811357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.803357 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:14:11.894304 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.892757 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:14:11.954321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.945440 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:11.954321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:11.946172 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:14:12.344309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:12.329340 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:14:12.440240 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:12.440210 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:14:12.828718 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:12.828682 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:12.959819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:12.959726 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:14:12.959819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:12.959786 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:13.012957 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.012920 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-xks-ca\") pod \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " Nov 02 20:14:13.013147 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.013098 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb09c2e-faa8-4231-afcb-6b3e08998ead-kube-api-access-tfwl2\" (UniqueName: \"kubernetes.io/projected/ddb09c2e-faa8-4231-afcb-6b3e08998ead-kube-api-access-tfwl2\") pod \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " Nov 02 20:14:13.013218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.013204 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-openshift-ca\") pod \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\" (UID: \"ddb09c2e-faa8-4231-afcb-6b3e08998ead\") " Nov 02 20:14:13.013620 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.013590 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-openshift-ca" pod "ddb09c2e-faa8-4231-afcb-6b3e08998ead" (UID: "ddb09c2e-faa8-4231-afcb-6b3e08998ead"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:13.013764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.013750 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:13.015195 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.015174 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-xks-ca" pod "ddb09c2e-faa8-4231-afcb-6b3e08998ead" (UID: "ddb09c2e-faa8-4231-afcb-6b3e08998ead"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:13.015729 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.015696 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb09c2e-faa8-4231-afcb-6b3e08998ead-kube-api-access-tfwl2" pod "ddb09c2e-faa8-4231-afcb-6b3e08998ead" (UID: "ddb09c2e-faa8-4231-afcb-6b3e08998ead"). InnerVolumeSpecName "kube-api-access-tfwl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:13.115150 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.115109 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfwl2\" (UniqueName: \"kubernetes.io/projected/ddb09c2e-faa8-4231-afcb-6b3e08998ead-kube-api-access-tfwl2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:13.115150 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.115152 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb09c2e-faa8-4231-afcb-6b3e08998ead-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:13.217637 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.217545 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg"] Nov 02 20:14:13.218475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.218454 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ddb09c2e-faa8-4231-afcb-6b3e08998ead" containerName="revoke-keys" Nov 02 20:14:13.218475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.218472 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb09c2e-faa8-4231-afcb-6b3e08998ead" containerName="revoke-keys" Nov 02 20:14:13.218680 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.218601 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb09c2e-faa8-4231-afcb-6b3e08998ead" containerName="revoke-keys" Nov 02 20:14:13.222456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.222421 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.281005 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.280939 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg"] Nov 02 20:14:13.319648 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.319613 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkwq\" (UniqueName: \"kubernetes.io/projected/d2ab431d-ed05-470d-8475-99df61005fe4-kube-api-access-4gkwq\") pod \"maas-api-revoke-sub-6ed60304-2m9sg\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.319834 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.319761 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed60304-2m9sg\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.319834 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.319799 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed60304-2m9sg\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.421498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.421463 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkwq\" (UniqueName: \"kubernetes.io/projected/d2ab431d-ed05-470d-8475-99df61005fe4-kube-api-access-4gkwq\") pod \"maas-api-revoke-sub-6ed60304-2m9sg\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.421924 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.421575 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed60304-2m9sg\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.421924 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.421618 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed60304-2m9sg\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.422262 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.422236 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed60304-2m9sg\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.424145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.424121 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed60304-2m9sg\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.474880 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.474805 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkwq\" (UniqueName: \"kubernetes.io/projected/d2ab431d-ed05-470d-8475-99df61005fe4-kube-api-access-4gkwq\") pod \"maas-api-revoke-sub-6ed60304-2m9sg\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.533894 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.533855 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" event={"ID":"ddb09c2e-faa8-4231-afcb-6b3e08998ead","Type":"ContainerDied","Data":"18f4cbc2856e6dd7ec8b99064fad81694992121e57ba61ac095a4741adea4339"} Nov 02 20:14:13.534075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.533922 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f4cbc2856e6dd7ec8b99064fad81694992121e57ba61ac095a4741adea4339" Nov 02 20:14:13.534075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.533995 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9" Nov 02 20:14:13.542461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.542420 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:13.932486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.932451 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:13.932907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.932889 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:13.933684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.933667 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:14:13.952699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:13.952659 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg"] Nov 02 20:14:13.953115 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:14:13.953091 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ab431d_ed05_470d_8475_99df61005fe4.slice/crio-f0b4023f33f262da029cfe476a5e44ba5808fe3a9a24319926908f62f3227a2f WatchSource:0}: Error finding container f0b4023f33f262da029cfe476a5e44ba5808fe3a9a24319926908f62f3227a2f: Status 404 returned error can't find the container with id f0b4023f33f262da029cfe476a5e44ba5808fe3a9a24319926908f62f3227a2f Nov 02 20:14:14.542780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:14.542745 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" event={"ID":"d2ab431d-ed05-470d-8475-99df61005fe4","Type":"ContainerStarted","Data":"cb460719902d3f74817de8c7181fdc4281050218a3f585664dc7663833d1f418"} Nov 02 20:14:14.543176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:14.542803 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" event={"ID":"d2ab431d-ed05-470d-8475-99df61005fe4","Type":"ContainerStarted","Data":"f0b4023f33f262da029cfe476a5e44ba5808fe3a9a24319926908f62f3227a2f"} Nov 02 20:14:14.651268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:14.651166 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" podStartSLOduration=1.651152051 podStartE2EDuration="1.651152051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:14.649027636 +0000 UTC m=+1297.259679726" watchObservedRunningTime="2026-11-02 20:14:14.651152051 +0000 UTC m=+1297.261804141" Nov 02 20:14:15.550938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:15.550895 5399 generic.go:356] "Generic (PLEG): container finished" podID="d2ab431d-ed05-470d-8475-99df61005fe4" containerID="cb460719902d3f74817de8c7181fdc4281050218a3f585664dc7663833d1f418" exitCode=0 Nov 02 20:14:15.551349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:15.551000 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" event={"ID":"d2ab431d-ed05-470d-8475-99df61005fe4","Type":"ContainerDied","Data":"cb460719902d3f74817de8c7181fdc4281050218a3f585664dc7663833d1f418"} Nov 02 20:14:16.729989 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:16.729965 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:16.765472 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:16.765415 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-xks-ca\") pod \"d2ab431d-ed05-470d-8475-99df61005fe4\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " Nov 02 20:14:16.765663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:16.765556 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ab431d-ed05-470d-8475-99df61005fe4-kube-api-access-4gkwq\" (UniqueName: \"kubernetes.io/projected/d2ab431d-ed05-470d-8475-99df61005fe4-kube-api-access-4gkwq\") pod \"d2ab431d-ed05-470d-8475-99df61005fe4\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " Nov 02 20:14:16.765663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:16.765643 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-openshift-ca\") pod \"d2ab431d-ed05-470d-8475-99df61005fe4\" (UID: \"d2ab431d-ed05-470d-8475-99df61005fe4\") " Nov 02 20:14:16.766027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:16.765978 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-openshift-ca" pod "d2ab431d-ed05-470d-8475-99df61005fe4" (UID: "d2ab431d-ed05-470d-8475-99df61005fe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:16.766269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:16.766245 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:16.767914 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:16.767881 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-xks-ca" pod "d2ab431d-ed05-470d-8475-99df61005fe4" (UID: "d2ab431d-ed05-470d-8475-99df61005fe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:16.768022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:16.767982 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ab431d-ed05-470d-8475-99df61005fe4-kube-api-access-4gkwq" pod "d2ab431d-ed05-470d-8475-99df61005fe4" (UID: "d2ab431d-ed05-470d-8475-99df61005fe4"). InnerVolumeSpecName "kube-api-access-4gkwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:16.867386 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:16.867270 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gkwq\" (UniqueName: \"kubernetes.io/projected/d2ab431d-ed05-470d-8475-99df61005fe4-kube-api-access-4gkwq\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:16.867386 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:16.867333 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ab431d-ed05-470d-8475-99df61005fe4-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:17.566336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:17.566310 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" Nov 02 20:14:17.566521 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:17.566306 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg" event={"ID":"d2ab431d-ed05-470d-8475-99df61005fe4","Type":"ContainerDied","Data":"f0b4023f33f262da029cfe476a5e44ba5808fe3a9a24319926908f62f3227a2f"} Nov 02 20:14:17.566521 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:17.566435 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0b4023f33f262da029cfe476a5e44ba5808fe3a9a24319926908f62f3227a2f" Nov 02 20:14:18.243479 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:14:18.243442 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:14:19.849643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.849607 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5"] Nov 02 20:14:19.850207 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.850194 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d2ab431d-ed05-470d-8475-99df61005fe4" containerName="revoke-keys" Nov 02 20:14:19.850292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.850208 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ab431d-ed05-470d-8475-99df61005fe4" containerName="revoke-keys" Nov 02 20:14:19.850390 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.850378 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ab431d-ed05-470d-8475-99df61005fe4" containerName="revoke-keys" Nov 02 20:14:19.854009 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.853993 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:19.906308 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.906265 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5"] Nov 02 20:14:19.908894 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.908868 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba11d4ae-6qwp5\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:19.909033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.909018 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd49g\" (UniqueName: \"kubernetes.io/projected/251aa06c-6ddf-4d9c-afd2-134515a1becb-kube-api-access-xd49g\") pod \"maas-api-revoke-sub-ba11d4ae-6qwp5\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:19.909105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.909091 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba11d4ae-6qwp5\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:19.947920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.947890 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:14:19.948049 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:19.947971 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:14:20.010749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:20.010712 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd49g\" (UniqueName: \"kubernetes.io/projected/251aa06c-6ddf-4d9c-afd2-134515a1becb-kube-api-access-xd49g\") pod \"maas-api-revoke-sub-ba11d4ae-6qwp5\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:20.010923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:20.010807 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba11d4ae-6qwp5\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:20.010923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:20.010876 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba11d4ae-6qwp5\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:20.012195 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:20.012159 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba11d4ae-6qwp5\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:20.014191 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:20.014132 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba11d4ae-6qwp5\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:20.063800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:20.063765 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd49g\" (UniqueName: \"kubernetes.io/projected/251aa06c-6ddf-4d9c-afd2-134515a1becb-kube-api-access-xd49g\") pod \"maas-api-revoke-sub-ba11d4ae-6qwp5\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:20.167120 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:20.167035 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:20.537745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:20.537709 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5"] Nov 02 20:14:20.537885 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:14:20.537741 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251aa06c_6ddf_4d9c_afd2_134515a1becb.slice/crio-0ab3384286ddd84801cc802313a6d29cae906ebef3b760e34bde58ef07873506 WatchSource:0}: Error finding container 0ab3384286ddd84801cc802313a6d29cae906ebef3b760e34bde58ef07873506: Status 404 returned error can't find the container with id 0ab3384286ddd84801cc802313a6d29cae906ebef3b760e34bde58ef07873506 Nov 02 20:14:20.589014 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:20.588977 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" event={"ID":"251aa06c-6ddf-4d9c-afd2-134515a1becb","Type":"ContainerStarted","Data":"0ab3384286ddd84801cc802313a6d29cae906ebef3b760e34bde58ef07873506"} Nov 02 20:14:21.598097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:21.598061 5399 generic.go:356] "Generic (PLEG): container finished" podID="251aa06c-6ddf-4d9c-afd2-134515a1becb" containerID="5e07fc1b36f58272d530202d39bcfe3c10c21e8d9c8442beed8e606fce57df6b" exitCode=0 Nov 02 20:14:21.598531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:21.598154 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" event={"ID":"251aa06c-6ddf-4d9c-afd2-134515a1becb","Type":"ContainerDied","Data":"5e07fc1b36f58272d530202d39bcfe3c10c21e8d9c8442beed8e606fce57df6b"} Nov 02 20:14:22.755387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:22.755360 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:22.848730 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:22.848699 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-openshift-ca\") pod \"251aa06c-6ddf-4d9c-afd2-134515a1becb\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " Nov 02 20:14:22.848912 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:22.848831 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251aa06c-6ddf-4d9c-afd2-134515a1becb-kube-api-access-xd49g\" (UniqueName: \"kubernetes.io/projected/251aa06c-6ddf-4d9c-afd2-134515a1becb-kube-api-access-xd49g\") pod \"251aa06c-6ddf-4d9c-afd2-134515a1becb\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " Nov 02 20:14:22.849000 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:22.848982 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-xks-ca\") pod \"251aa06c-6ddf-4d9c-afd2-134515a1becb\" (UID: \"251aa06c-6ddf-4d9c-afd2-134515a1becb\") " Nov 02 20:14:22.849123 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:22.849100 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-openshift-ca" pod "251aa06c-6ddf-4d9c-afd2-134515a1becb" (UID: "251aa06c-6ddf-4d9c-afd2-134515a1becb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:22.849523 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:22.849492 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:22.850971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:22.850942 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251aa06c-6ddf-4d9c-afd2-134515a1becb-kube-api-access-xd49g" pod "251aa06c-6ddf-4d9c-afd2-134515a1becb" (UID: "251aa06c-6ddf-4d9c-afd2-134515a1becb"). InnerVolumeSpecName "kube-api-access-xd49g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:22.851074 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:22.850947 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-xks-ca" pod "251aa06c-6ddf-4d9c-afd2-134515a1becb" (UID: "251aa06c-6ddf-4d9c-afd2-134515a1becb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:22.950395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:22.950365 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd49g\" (UniqueName: \"kubernetes.io/projected/251aa06c-6ddf-4d9c-afd2-134515a1becb-kube-api-access-xd49g\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:22.950395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:22.950393 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251aa06c-6ddf-4d9c-afd2-134515a1becb-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:23.613629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:23.613591 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" event={"ID":"251aa06c-6ddf-4d9c-afd2-134515a1becb","Type":"ContainerDied","Data":"0ab3384286ddd84801cc802313a6d29cae906ebef3b760e34bde58ef07873506"} Nov 02 20:14:23.613793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:23.613644 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ab3384286ddd84801cc802313a6d29cae906ebef3b760e34bde58ef07873506" Nov 02 20:14:23.613793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:23.613600 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5" Nov 02 20:14:26.044159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:26.044125 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5"] Nov 02 20:14:26.099543 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:26.099512 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79c10ee9-mdhb5"] Nov 02 20:14:27.510407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:27.510374 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb9c8142-0f46-4599-9aa0-31a2d62100ec" path="/var/lib/kubelet/pods/bb9c8142-0f46-4599-9aa0-31a2d62100ec/volumes" Nov 02 20:14:31.538058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:31.538024 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:14:32.001312 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.001235 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46"] Nov 02 20:14:32.002268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.002242 5399 cpu_manager.go:414] "containerMap: removing container" podUID="251aa06c-6ddf-4d9c-afd2-134515a1becb" containerName="revoke-keys" Nov 02 20:14:32.002268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.002267 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251aa06c-6ddf-4d9c-afd2-134515a1becb" containerName="revoke-keys" Nov 02 20:14:32.002485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.002469 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="251aa06c-6ddf-4d9c-afd2-134515a1becb" containerName="revoke-keys" Nov 02 20:14:32.006577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.006552 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:32.056964 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.056930 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46"] Nov 02 20:14:32.097640 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.097590 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:14:32.097820 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.097706 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:14:32.190251 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.190213 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnts7\" (UniqueName: \"kubernetes.io/projected/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-kube-api-access-tnts7\") pod \"maas-api-revoke-sub-b6fc5f80-q7z46\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:32.190431 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.190400 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6fc5f80-q7z46\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:32.190431 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.190426 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6fc5f80-q7z46\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:32.292853 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.292733 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnts7\" (UniqueName: \"kubernetes.io/projected/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-kube-api-access-tnts7\") pod \"maas-api-revoke-sub-b6fc5f80-q7z46\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:32.293104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.292895 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6fc5f80-q7z46\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:32.293104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.292922 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6fc5f80-q7z46\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:32.313296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.294536 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6fc5f80-q7z46\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:32.339296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.318577 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6fc5f80-q7z46\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:32.376618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.376551 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnts7\" (UniqueName: \"kubernetes.io/projected/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-kube-api-access-tnts7\") pod \"maas-api-revoke-sub-b6fc5f80-q7z46\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:32.624114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:32.624071 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:33.527726 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.373577 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn"] Nov 02 20:14:33.746475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.746431 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.748046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.748024 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr"] Nov 02 20:14:33.851302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.818948 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:33.851762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.831219 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:33.851997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.851975 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.852169 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.852150 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.852335 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.852317 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.852507 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.852488 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.852639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.852622 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:33.852782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.852763 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:33.852898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.852882 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:33.853020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.853005 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkx2x\" (UniqueName: \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-kube-api-access-gkx2x\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.878416 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.878371 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:33.878756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.878721 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:33.878925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.878905 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7b7\" (UniqueName: \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-kube-api-access-xs7b7\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:33.879036 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.879020 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.879162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.879145 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.879315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.879297 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.879508 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.879489 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.879650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.879635 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:33.888302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.879807 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:33.996683 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.996643 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.996966 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.996943 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:33.997106 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:33.997090 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.001392 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.001362 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.001584 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.001566 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.001722 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.001707 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.026405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.026367 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.026622 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.026605 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkx2x\" (UniqueName: \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-kube-api-access-gkx2x\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.026773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.026754 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.027015 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.026996 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.027132 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.027115 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7b7\" (UniqueName: \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-kube-api-access-xs7b7\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.031298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.027240 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.035336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.031474 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.049313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.035514 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.049649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.049613 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.060300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.049809 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.084298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.060593 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.084640 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.084611 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.084831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.032616 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.085719 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.027359 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.085884 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.028650 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.086030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.007866 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.086179 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.054092 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.086338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.027080 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.086469 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.054684 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.086584 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.029690 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.086702 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.033093 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.086828 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.085255 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.149348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.139795 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.149348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.146001 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.174307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.158382 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.235326 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.205938 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 20:14:34.263649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.263189 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:34.338299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.316663 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:34.338299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.318666 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:34.413726 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.413613 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:14:34.518300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.510493 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:14:34.628895 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.583799 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn"] Nov 02 20:14:34.992594 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.992566 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr"] Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.626475 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.783833 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.784033 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7b7\" (UniqueName: \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-kube-api-access-xs7b7\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.805201 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.827525 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.827781 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.828467 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkx2x\" (UniqueName: \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-kube-api-access-gkx2x\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.913676 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:34.597444 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.017036 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.017360 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46"] Nov 02 20:14:35.018349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.017516 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" event={"ID":"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae","Type":"ContainerStarted","Data":"0ce9ce5f04e6a77cdaf94267492c43eaa83a61ecf83d2ba4cc7f249c097c80c8"} Nov 02 20:14:35.226346 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.221375 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.226346 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.221801 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:14:35.226346 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.223999 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:35.300788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.299540 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:35.324313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.321426 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.324313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.321871 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:14:35.473563 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.435838 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.473563 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.436206 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:14:35.530404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:35.530360 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:35.781385 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:14:35.729315 5399 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762b1976_bfc9_4e3e_a44e_a5c8c65f7bae.slice/crio-f6f219ad423ba67dc72f933ee107f7a04833c1881dd9d528c5a7fca962f8ee23.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:14:36.022527 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.022480 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.050347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.024176 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:14:36.051344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.036945 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.052673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.036982 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:14:36.061094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.037003 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.185253 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.185212 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:14:36.209492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.209447 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:14:36.209778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.209760 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:14:36.209997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.209968 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:14:36.210162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.210148 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:14:36.210360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.210330 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:14:36.238298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.234916 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.345310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.328615 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:14:36.459746 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.459706 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:14:36.534570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.534503 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:14:36.867368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.863951 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:14:36.878816 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.878196 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:14:36.881513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.881224 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:14:36.885647 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.885418 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:14:36.893549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.892770 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:14:36.894156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.894135 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:14:36.903247 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.897254 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:14:36.903247 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.902242 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:14:36.904813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:36.904795 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:14:37.168007 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:14:37.167979 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f04fe12_86a0_479c_b0c0_2c9df43f3fc5.slice/crio-64d9fb4522ab49c0ca8436a093307f984c9c1bd9d3324065b8d38caeee1ab195 WatchSource:0}: Error finding container 64d9fb4522ab49c0ca8436a093307f984c9c1bd9d3324065b8d38caeee1ab195: Status 404 returned error can't find the container with id 64d9fb4522ab49c0ca8436a093307f984c9c1bd9d3324065b8d38caeee1ab195 Nov 02 20:14:37.169564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.169540 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn"] Nov 02 20:14:37.170930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.170889 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:37.171027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.170971 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:37.171027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.171010 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:37.278498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.278467 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr"] Nov 02 20:14:37.303110 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:14:37.303070 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7abb95_84f1_4a34_a6b3_403a5d02bd1d.slice/crio-7cf7c593d9803c504a3f90e80c3e3de41f61526a0a2e17ef312d21374a2e088a WatchSource:0}: Error finding container 7cf7c593d9803c504a3f90e80c3e3de41f61526a0a2e17ef312d21374a2e088a: Status 404 returned error can't find the container with id 7cf7c593d9803c504a3f90e80c3e3de41f61526a0a2e17ef312d21374a2e088a Nov 02 20:14:37.313166 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.312076 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:37.313166 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.312158 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:37.313166 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.312198 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:37.521774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.521653 5399 generic.go:356] "Generic (PLEG): container finished" podID="762b1976-bfc9-4e3e-a44e-a5c8c65f7bae" containerID="f6f219ad423ba67dc72f933ee107f7a04833c1881dd9d528c5a7fca962f8ee23" exitCode=0 Nov 02 20:14:37.521774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.521724 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" event={"ID":"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae","Type":"ContainerDied","Data":"f6f219ad423ba67dc72f933ee107f7a04833c1881dd9d528c5a7fca962f8ee23"} Nov 02 20:14:37.523980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.523897 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" event={"ID":"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d","Type":"ContainerStarted","Data":"c63567174de3f5f24853e05c5c3a982589e84a8612bf453cb0ee9bfba7a09071"} Nov 02 20:14:37.523980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.523948 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" event={"ID":"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d","Type":"ContainerStarted","Data":"7cf7c593d9803c504a3f90e80c3e3de41f61526a0a2e17ef312d21374a2e088a"} Nov 02 20:14:37.527224 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.527194 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" event={"ID":"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5","Type":"ContainerStarted","Data":"cd997f825b4682dd93187ab4872aebf0127904aeba014c9eefa79089428fdfdf"} Nov 02 20:14:37.527457 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:37.527437 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" event={"ID":"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5","Type":"ContainerStarted","Data":"64d9fb4522ab49c0ca8436a093307f984c9c1bd9d3324065b8d38caeee1ab195"} Nov 02 20:14:38.068328 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.068291 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt"] Nov 02 20:14:38.076573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.076532 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e379a6bf-ngbjt"] Nov 02 20:14:38.225156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.225121 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:38.231333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.226263 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:14:38.231583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.231553 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:14:38.353358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.352366 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:38.379335 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.377578 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:14:38.379335 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.377638 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:14:38.482828 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.480792 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" podStartSLOduration=5.480765212 podStartE2EDuration="5.480765212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:38.455153166 +0000 UTC m=+1321.065805259" watchObservedRunningTime="2026-11-02 20:14:38.480765212 +0000 UTC m=+1321.091417304" Nov 02 20:14:38.683355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.674181 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" podStartSLOduration=5.674162004 podStartE2EDuration="5.674162004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:38.672799998 +0000 UTC m=+1321.283452089" watchObservedRunningTime="2026-11-02 20:14:38.674162004 +0000 UTC m=+1321.284814099" Nov 02 20:14:38.994322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:38.994066 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:39.123727 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.123696 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-xks-ca\") pod \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " Nov 02 20:14:39.124067 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.123761 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-kube-api-access-tnts7\" (UniqueName: \"kubernetes.io/projected/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-kube-api-access-tnts7\") pod \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " Nov 02 20:14:39.124157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.124078 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-openshift-ca\") pod \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\" (UID: \"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae\") " Nov 02 20:14:39.124451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.124413 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-openshift-ca" pod "762b1976-bfc9-4e3e-a44e-a5c8c65f7bae" (UID: "762b1976-bfc9-4e3e-a44e-a5c8c65f7bae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:39.124845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.124824 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:39.155477 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.152409 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-xks-ca" pod "762b1976-bfc9-4e3e-a44e-a5c8c65f7bae" (UID: "762b1976-bfc9-4e3e-a44e-a5c8c65f7bae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:39.188303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.185495 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-kube-api-access-tnts7" pod "762b1976-bfc9-4e3e-a44e-a5c8c65f7bae" (UID: "762b1976-bfc9-4e3e-a44e-a5c8c65f7bae"). InnerVolumeSpecName "kube-api-access-tnts7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:39.345728 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.345098 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:14:39.345728 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.345159 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:14:39.362297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.351523 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:14:39.362297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.351609 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:14:39.413298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.409413 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:39.413298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.409479 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnts7\" (UniqueName: \"kubernetes.io/projected/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae-kube-api-access-tnts7\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:39.520870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.520817 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f79054-8e32-49e9-9865-a00889f49002" path="/var/lib/kubelet/pods/c7f79054-8e32-49e9-9865-a00889f49002/volumes" Nov 02 20:14:39.600368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.600262 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" Nov 02 20:14:39.600745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.600679 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46" event={"ID":"762b1976-bfc9-4e3e-a44e-a5c8c65f7bae","Type":"ContainerDied","Data":"0ce9ce5f04e6a77cdaf94267492c43eaa83a61ecf83d2ba4cc7f249c097c80c8"} Nov 02 20:14:39.600954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:39.600935 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ce9ce5f04e6a77cdaf94267492c43eaa83a61ecf83d2ba4cc7f249c097c80c8" Nov 02 20:14:40.225444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:40.225411 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:14:40.225642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:40.225458 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:14:40.351686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:40.351650 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:14:40.352255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:40.352222 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:14:41.225604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.225576 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:14:41.225810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.225789 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:14:41.349296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.348379 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:14:41.349296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.348438 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:14:41.513636 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.513559 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr"] Nov 02 20:14:41.514151 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.514138 5399 cpu_manager.go:414] "containerMap: removing container" podUID="762b1976-bfc9-4e3e-a44e-a5c8c65f7bae" containerName="revoke-keys" Nov 02 20:14:41.514199 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.514152 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762b1976-bfc9-4e3e-a44e-a5c8c65f7bae" containerName="revoke-keys" Nov 02 20:14:41.514313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.514303 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="762b1976-bfc9-4e3e-a44e-a5c8c65f7bae" containerName="revoke-keys" Nov 02 20:14:41.518030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.518010 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:41.591223 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.591185 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr"] Nov 02 20:14:41.617618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.617588 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:14:41.617618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.617613 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:14:41.702193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.702149 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189c82ec-t7mrr\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:41.702424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.702208 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189c82ec-t7mrr\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:41.702481 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.702416 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjdb\" (UniqueName: \"kubernetes.io/projected/6c06fb08-a82b-4c43-924a-253cc7fec830-kube-api-access-bvjdb\") pod \"maas-api-revoke-sub-189c82ec-t7mrr\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:41.803436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.803353 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189c82ec-t7mrr\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:41.803436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.803399 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189c82ec-t7mrr\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:41.803663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.803460 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjdb\" (UniqueName: \"kubernetes.io/projected/6c06fb08-a82b-4c43-924a-253cc7fec830-kube-api-access-bvjdb\") pod \"maas-api-revoke-sub-189c82ec-t7mrr\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:41.804329 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.804306 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189c82ec-t7mrr\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:41.805776 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.805756 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189c82ec-t7mrr\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:41.886725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:41.886600 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjdb\" (UniqueName: \"kubernetes.io/projected/6c06fb08-a82b-4c43-924a-253cc7fec830-kube-api-access-bvjdb\") pod \"maas-api-revoke-sub-189c82ec-t7mrr\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:42.180464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:42.179740 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:42.225343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:42.225313 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:14:42.225486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:42.225369 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:14:42.335314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:42.331651 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:14:42.335314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:42.331723 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:14:42.605471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:42.605439 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr"] Nov 02 20:14:42.606042 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:14:42.606016 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c06fb08_a82b_4c43_924a_253cc7fec830.slice/crio-6dde002c751b991d049d1de1487710a41b404b0cfaef9a37ff592e1280c812cc WatchSource:0}: Error finding container 6dde002c751b991d049d1de1487710a41b404b0cfaef9a37ff592e1280c812cc: Status 404 returned error can't find the container with id 6dde002c751b991d049d1de1487710a41b404b0cfaef9a37ff592e1280c812cc Nov 02 20:14:42.678795 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:42.678735 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" event={"ID":"6c06fb08-a82b-4c43-924a-253cc7fec830","Type":"ContainerStarted","Data":"6dde002c751b991d049d1de1487710a41b404b0cfaef9a37ff592e1280c812cc"} Nov 02 20:14:43.225607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:43.225525 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:14:43.225607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:43.225590 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:14:43.364307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:43.363976 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:14:43.364307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:43.364049 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:14:43.690846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:43.690803 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" event={"ID":"6c06fb08-a82b-4c43-924a-253cc7fec830","Type":"ContainerStarted","Data":"83e1a6fe19f02f5b8abbc4bf553b5d6570398a8b400b570ce086aa3dd4bc602b"} Nov 02 20:14:44.225464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:44.225427 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:14:44.225638 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:44.225495 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:14:44.330022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:44.329987 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:14:44.330167 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:44.330034 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:14:44.705083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:44.705042 5399 generic.go:356] "Generic (PLEG): container finished" podID="6c06fb08-a82b-4c43-924a-253cc7fec830" containerID="83e1a6fe19f02f5b8abbc4bf553b5d6570398a8b400b570ce086aa3dd4bc602b" exitCode=0 Nov 02 20:14:44.705624 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:44.705129 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" event={"ID":"6c06fb08-a82b-4c43-924a-253cc7fec830","Type":"ContainerDied","Data":"83e1a6fe19f02f5b8abbc4bf553b5d6570398a8b400b570ce086aa3dd4bc602b"} Nov 02 20:14:45.230089 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:45.230055 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:45.230505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:45.230485 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:45.231344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:45.231324 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:14:45.333652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:45.333623 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:45.333951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:45.333926 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:45.334586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:45.334569 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:14:45.869030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:45.869004 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:46.000155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.000054 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-xks-ca\") pod \"6c06fb08-a82b-4c43-924a-253cc7fec830\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " Nov 02 20:14:46.000155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.000130 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c06fb08-a82b-4c43-924a-253cc7fec830-kube-api-access-bvjdb\" (UniqueName: \"kubernetes.io/projected/6c06fb08-a82b-4c43-924a-253cc7fec830-kube-api-access-bvjdb\") pod \"6c06fb08-a82b-4c43-924a-253cc7fec830\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " Nov 02 20:14:46.000411 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.000224 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-openshift-ca\") pod \"6c06fb08-a82b-4c43-924a-253cc7fec830\" (UID: \"6c06fb08-a82b-4c43-924a-253cc7fec830\") " Nov 02 20:14:46.000946 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.000909 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-openshift-ca" pod "6c06fb08-a82b-4c43-924a-253cc7fec830" (UID: "6c06fb08-a82b-4c43-924a-253cc7fec830"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:46.002358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.002333 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-xks-ca" pod "6c06fb08-a82b-4c43-924a-253cc7fec830" (UID: "6c06fb08-a82b-4c43-924a-253cc7fec830"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:46.002849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.002828 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c06fb08-a82b-4c43-924a-253cc7fec830-kube-api-access-bvjdb" pod "6c06fb08-a82b-4c43-924a-253cc7fec830" (UID: "6c06fb08-a82b-4c43-924a-253cc7fec830"). InnerVolumeSpecName "kube-api-access-bvjdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:46.101873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.101840 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:46.101873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.101867 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvjdb\" (UniqueName: \"kubernetes.io/projected/6c06fb08-a82b-4c43-924a-253cc7fec830-kube-api-access-bvjdb\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:46.101873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.101877 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c06fb08-a82b-4c43-924a-253cc7fec830-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:46.722963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.722933 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" Nov 02 20:14:46.723132 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.722963 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr" event={"ID":"6c06fb08-a82b-4c43-924a-253cc7fec830","Type":"ContainerDied","Data":"6dde002c751b991d049d1de1487710a41b404b0cfaef9a37ff592e1280c812cc"} Nov 02 20:14:46.723132 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.723012 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dde002c751b991d049d1de1487710a41b404b0cfaef9a37ff592e1280c812cc" Nov 02 20:14:46.998990 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.998900 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs"] Nov 02 20:14:46.999569 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.999554 5399 cpu_manager.go:414] "containerMap: removing container" podUID="6c06fb08-a82b-4c43-924a-253cc7fec830" containerName="revoke-keys" Nov 02 20:14:46.999611 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.999570 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c06fb08-a82b-4c43-924a-253cc7fec830" containerName="revoke-keys" Nov 02 20:14:46.999686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:46.999676 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c06fb08-a82b-4c43-924a-253cc7fec830" containerName="revoke-keys" Nov 02 20:14:47.003568 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.003546 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.095388 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.095352 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs"] Nov 02 20:14:47.117387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.117351 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.117557 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.117481 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.117650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.117573 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.117650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.117635 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.117766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.117665 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.117766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.117694 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.117869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.117809 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.117869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.117827 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49bj\" (UniqueName: \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-kube-api-access-j49bj\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.117976 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.117888 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.218802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.218765 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.218987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.218865 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.218987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.218949 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219077 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219012 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219077 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219057 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219186 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219162 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219244 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219210 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219237 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219352 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219382 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49bj\" (UniqueName: \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-kube-api-access-j49bj\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219466 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219749 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219857 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219799 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.219857 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.219803 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.221467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.221445 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.221689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.221670 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.272214 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.272135 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.272214 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.272164 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49bj\" (UniqueName: \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-kube-api-access-j49bj\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.316817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.316777 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:47.741217 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:14:47.741177 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ab73d4_d368_4b9f_aff9_29f2711cb7fb.slice/crio-e1463102f0bd14c09c73af3892d75cc335cf8138c8b5ba3f37492610f1be8251 WatchSource:0}: Error finding container e1463102f0bd14c09c73af3892d75cc335cf8138c8b5ba3f37492610f1be8251: Status 404 returned error can't find the container with id e1463102f0bd14c09c73af3892d75cc335cf8138c8b5ba3f37492610f1be8251 Nov 02 20:14:47.742544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.742511 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs"] Nov 02 20:14:47.743464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.743430 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:47.743571 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.743503 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:47.743571 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:47.743547 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:48.241632 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:14:48.241563 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:14:48.760609 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:48.760570 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" event={"ID":"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb","Type":"ContainerStarted","Data":"f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426"} Nov 02 20:14:48.760824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:48.760634 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" event={"ID":"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb","Type":"ContainerStarted","Data":"e1463102f0bd14c09c73af3892d75cc335cf8138c8b5ba3f37492610f1be8251"} Nov 02 20:14:48.917901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:48.917847 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" podStartSLOduration=2.917834876 podStartE2EDuration="2.917834876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:48.914742554 +0000 UTC m=+1331.525394658" watchObservedRunningTime="2026-11-02 20:14:48.917834876 +0000 UTC m=+1331.528486966" Nov 02 20:14:49.317693 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:49.317659 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:49.318929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:49.318908 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:14:49.319001 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:49.318961 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:14:50.317266 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:50.317229 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:14:50.317472 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:50.317310 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:14:50.986735 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:50.986698 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7198-f080ff"] Nov 02 20:14:50.991323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:50.991299 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:14:51.057850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:51.057811 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-f080ff"] Nov 02 20:14:51.094078 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:51.094050 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-28p54\"" Nov 02 20:14:51.188163 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:51.188127 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npdmc\" (UniqueName: \"kubernetes.io/projected/e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4-kube-api-access-npdmc\") pod \"test-curl-7198-f080ff\" (UID: \"e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4\") " pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:14:51.291757 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:51.290671 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npdmc\" (UniqueName: \"kubernetes.io/projected/e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4-kube-api-access-npdmc\") pod \"test-curl-7198-f080ff\" (UID: \"e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4\") " pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:14:51.317125 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:51.317093 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:14:51.317361 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:51.317150 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:14:51.362617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:51.362583 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npdmc\" (UniqueName: \"kubernetes.io/projected/e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4-kube-api-access-npdmc\") pod \"test-curl-7198-f080ff\" (UID: \"e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4\") " pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:14:51.604465 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:51.604424 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:14:52.001255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:52.001225 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-f080ff"] Nov 02 20:14:52.002173 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:14:52.002148 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5c6faf9_b9be_4069_9d3b_1440d1d1b2f4.slice/crio-3c24799e51c03377a5dd65dd4651511afd3fd1a5d21dc46a8d7b828317772fcb WatchSource:0}: Error finding container 3c24799e51c03377a5dd65dd4651511afd3fd1a5d21dc46a8d7b828317772fcb: Status 404 returned error can't find the container with id 3c24799e51c03377a5dd65dd4651511afd3fd1a5d21dc46a8d7b828317772fcb Nov 02 20:14:52.004621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:52.004605 5399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:14:52.317101 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:52.317069 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:14:52.317321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:52.317124 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:14:52.790503 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:52.790424 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-f080ff" event={"ID":"e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4","Type":"ContainerStarted","Data":"3c24799e51c03377a5dd65dd4651511afd3fd1a5d21dc46a8d7b828317772fcb"} Nov 02 20:14:53.318048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:53.318014 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:14:53.318441 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:53.318063 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:14:54.317490 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.317453 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:14:54.317674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.317512 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:14:54.649505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.649406 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl"] Nov 02 20:14:54.654932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.654905 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:54.705985 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.705950 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl"] Nov 02 20:14:54.746135 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.746104 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:14:54.746372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.746314 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:14:54.767196 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.767156 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde7750e-lqxsl\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:54.767371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.767316 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde7750e-lqxsl\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:54.767461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.767448 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkgd\" (UniqueName: \"kubernetes.io/projected/21079db8-15e7-4e85-9582-c4863ccee643-kube-api-access-vdkgd\") pod \"maas-api-revoke-sub-fde7750e-lqxsl\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:54.874321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.869429 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde7750e-lqxsl\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:54.874321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.869570 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkgd\" (UniqueName: \"kubernetes.io/projected/21079db8-15e7-4e85-9582-c4863ccee643-kube-api-access-vdkgd\") pod \"maas-api-revoke-sub-fde7750e-lqxsl\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:54.874321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.869775 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde7750e-lqxsl\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:54.874321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.870630 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fde7750e-lqxsl\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:54.874321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.873550 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fde7750e-lqxsl\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:54.926817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.926710 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkgd\" (UniqueName: \"kubernetes.io/projected/21079db8-15e7-4e85-9582-c4863ccee643-kube-api-access-vdkgd\") pod \"maas-api-revoke-sub-fde7750e-lqxsl\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:54.974813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:54.974775 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:55.322640 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:55.322612 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:55.323201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:55.323173 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:55.323898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:55.323880 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:14:55.374661 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:14:55.374632 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21079db8_15e7_4e85_9582_c4863ccee643.slice/crio-e8bec7882d45bbfaf3c9c99d5e5308d4ba2e198d79537171afb073998857ea71 WatchSource:0}: Error finding container e8bec7882d45bbfaf3c9c99d5e5308d4ba2e198d79537171afb073998857ea71: Status 404 returned error can't find the container with id e8bec7882d45bbfaf3c9c99d5e5308d4ba2e198d79537171afb073998857ea71 Nov 02 20:14:55.375300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:55.375248 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl"] Nov 02 20:14:55.822584 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:55.822539 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-f080ff" event={"ID":"e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4","Type":"ContainerStarted","Data":"324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79"} Nov 02 20:14:55.824223 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:55.824194 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" event={"ID":"21079db8-15e7-4e85-9582-c4863ccee643","Type":"ContainerStarted","Data":"1a925c2a41d5e10381ad8bc9127b13ad172b69a6bb8cdac1b61f90553bed9a8d"} Nov 02 20:14:55.824355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:55.824252 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" event={"ID":"21079db8-15e7-4e85-9582-c4863ccee643","Type":"ContainerStarted","Data":"e8bec7882d45bbfaf3c9c99d5e5308d4ba2e198d79537171afb073998857ea71"} Nov 02 20:14:55.930100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:55.930046 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7198-f080ff" podStartSLOduration=2.637408051 podStartE2EDuration="5.93003213s" totalImagesPullingTime="3.292624079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:55.925535056 +0000 UTC m=+1338.536187148" watchObservedRunningTime="2026-11-02 20:14:55.93003213 +0000 UTC m=+1338.540684221" Nov 02 20:14:56.036338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:56.036224 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" podStartSLOduration=2.036203808 podStartE2EDuration="2.036203808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:56.032773746 +0000 UTC m=+1338.643425833" watchObservedRunningTime="2026-11-02 20:14:56.036203808 +0000 UTC m=+1338.646855899" Nov 02 20:14:56.832807 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:56.832769 5399 generic.go:356] "Generic (PLEG): container finished" podID="21079db8-15e7-4e85-9582-c4863ccee643" containerID="1a925c2a41d5e10381ad8bc9127b13ad172b69a6bb8cdac1b61f90553bed9a8d" exitCode=0 Nov 02 20:14:56.833209 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:56.832854 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" event={"ID":"21079db8-15e7-4e85-9582-c4863ccee643","Type":"ContainerDied","Data":"1a925c2a41d5e10381ad8bc9127b13ad172b69a6bb8cdac1b61f90553bed9a8d"} Nov 02 20:14:57.289996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.289911 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7198-f080ff"] Nov 02 20:14:57.295393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.295363 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7198-f080ff"] Nov 02 20:14:57.442400 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.442364 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7198-66c4c1"] Nov 02 20:14:57.443008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.442994 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" containerName="test-curl-7198-f080ff" Nov 02 20:14:57.443053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.443009 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" containerName="test-curl-7198-f080ff" Nov 02 20:14:57.443131 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.443122 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" containerName="test-curl-7198-f080ff" Nov 02 20:14:57.446790 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.446774 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-66c4c1" Nov 02 20:14:57.497156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.497127 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-66c4c1"] Nov 02 20:14:57.510348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.510308 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwq2\" (UniqueName: \"kubernetes.io/projected/2a028cdd-b2a9-4751-9c35-a916d21d1b10-kube-api-access-qzwq2\") pod \"test-curl-7198-66c4c1\" (UID: \"2a028cdd-b2a9-4751-9c35-a916d21d1b10\") " pod="opendatahub/test-curl-7198-66c4c1" Nov 02 20:14:57.621713 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.621264 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwq2\" (UniqueName: \"kubernetes.io/projected/2a028cdd-b2a9-4751-9c35-a916d21d1b10-kube-api-access-qzwq2\") pod \"test-curl-7198-66c4c1\" (UID: \"2a028cdd-b2a9-4751-9c35-a916d21d1b10\") " pod="opendatahub/test-curl-7198-66c4c1" Nov 02 20:14:57.674108 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.674069 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwq2\" (UniqueName: \"kubernetes.io/projected/2a028cdd-b2a9-4751-9c35-a916d21d1b10-kube-api-access-qzwq2\") pod \"test-curl-7198-66c4c1\" (UID: \"2a028cdd-b2a9-4751-9c35-a916d21d1b10\") " pod="opendatahub/test-curl-7198-66c4c1" Nov 02 20:14:57.759433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.759398 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-66c4c1" Nov 02 20:14:57.846326 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:57.842312 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7198-f080ff" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" containerName="test-curl-7198-f080ff" containerID="cri-o://324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79" gracePeriod=2 Nov 02 20:14:57.890214 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:14:57.890099 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-f080ff\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:14:58.010372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.010347 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:58.057800 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:14:58.057762 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-f080ff\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:14:58.127840 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.127799 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-66c4c1"] Nov 02 20:14:58.127997 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:14:58.127975 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a028cdd_b2a9_4751_9c35_a916d21d1b10.slice/crio-b76b546611df947fcf8bc62ebb00dcf78ff38e70a7baa3473f2edfb54f4a2f1f WatchSource:0}: Error finding container b76b546611df947fcf8bc62ebb00dcf78ff38e70a7baa3473f2edfb54f4a2f1f: Status 404 returned error can't find the container with id b76b546611df947fcf8bc62ebb00dcf78ff38e70a7baa3473f2edfb54f4a2f1f Nov 02 20:14:58.135851 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.135823 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-openshift-ca\") pod \"21079db8-15e7-4e85-9582-c4863ccee643\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " Nov 02 20:14:58.135961 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.135915 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-xks-ca\") pod \"21079db8-15e7-4e85-9582-c4863ccee643\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " Nov 02 20:14:58.136080 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.136058 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21079db8-15e7-4e85-9582-c4863ccee643-kube-api-access-vdkgd\" (UniqueName: \"kubernetes.io/projected/21079db8-15e7-4e85-9582-c4863ccee643-kube-api-access-vdkgd\") pod \"21079db8-15e7-4e85-9582-c4863ccee643\" (UID: \"21079db8-15e7-4e85-9582-c4863ccee643\") " Nov 02 20:14:58.136261 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.136236 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-openshift-ca" pod "21079db8-15e7-4e85-9582-c4863ccee643" (UID: "21079db8-15e7-4e85-9582-c4863ccee643"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:58.136630 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.136601 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:58.138058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.138036 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-xks-ca" pod "21079db8-15e7-4e85-9582-c4863ccee643" (UID: "21079db8-15e7-4e85-9582-c4863ccee643"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:58.138140 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.138122 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21079db8-15e7-4e85-9582-c4863ccee643-kube-api-access-vdkgd" pod "21079db8-15e7-4e85-9582-c4863ccee643" (UID: "21079db8-15e7-4e85-9582-c4863ccee643"). InnerVolumeSpecName "kube-api-access-vdkgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:58.238534 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.238505 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdkgd\" (UniqueName: \"kubernetes.io/projected/21079db8-15e7-4e85-9582-c4863ccee643-kube-api-access-vdkgd\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:58.238534 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.238531 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21079db8-15e7-4e85-9582-c4863ccee643-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:58.850652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.850613 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-66c4c1" event={"ID":"2a028cdd-b2a9-4751-9c35-a916d21d1b10","Type":"ContainerStarted","Data":"a7a6acf5478c028ee68ced84403e5bd72b52b5caf2feb81e278f264a58c931ee"} Nov 02 20:14:58.851057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.850663 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-66c4c1" event={"ID":"2a028cdd-b2a9-4751-9c35-a916d21d1b10","Type":"ContainerStarted","Data":"b76b546611df947fcf8bc62ebb00dcf78ff38e70a7baa3473f2edfb54f4a2f1f"} Nov 02 20:14:58.852151 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.852129 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" event={"ID":"21079db8-15e7-4e85-9582-c4863ccee643","Type":"ContainerDied","Data":"e8bec7882d45bbfaf3c9c99d5e5308d4ba2e198d79537171afb073998857ea71"} Nov 02 20:14:58.852265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.852168 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8bec7882d45bbfaf3c9c99d5e5308d4ba2e198d79537171afb073998857ea71" Nov 02 20:14:58.852265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.852138 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl" Nov 02 20:14:58.956002 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:14:58.955901 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7198-66c4c1" podStartSLOduration=1.955887786 podStartE2EDuration="1.955887786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:58.954449412 +0000 UTC m=+1341.565101497" watchObservedRunningTime="2026-11-02 20:14:58.955887786 +0000 UTC m=+1341.566539876" Nov 02 20:14:59.001651 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:14:59.001611 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-f080ff\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:14:59.048954 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:14:59.048913 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-f080ff\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:15:00.098750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.098684 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:15:00.146142 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:00.146105 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-f080ff\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:15:00.162228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.162183 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz"] Nov 02 20:15:00.162913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.162894 5399 cpu_manager.go:414] "containerMap: removing container" podUID="21079db8-15e7-4e85-9582-c4863ccee643" containerName="revoke-keys" Nov 02 20:15:00.162913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.162912 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21079db8-15e7-4e85-9582-c4863ccee643" containerName="revoke-keys" Nov 02 20:15:00.163061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.163053 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="21079db8-15e7-4e85-9582-c4863ccee643" containerName="revoke-keys" Nov 02 20:15:00.167601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.167578 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8"] Nov 02 20:15:00.167703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.167684 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" Nov 02 20:15:00.173028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.173004 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" Nov 02 20:15:00.198713 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.198676 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822"] Nov 02 20:15:00.203060 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.203037 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" Nov 02 20:15:00.215057 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:00.215027 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-f080ff\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:15:00.216063 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.216046 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:15:00.258363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.258330 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz"] Nov 02 20:15:00.264743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.264707 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8"] Nov 02 20:15:00.266970 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.266941 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4-kube-api-access-npdmc\" (UniqueName: \"kubernetes.io/projected/e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4-kube-api-access-npdmc\") pod \"e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4\" (UID: \"e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4\") " Nov 02 20:15:00.267322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.267259 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56l27\" (UniqueName: \"kubernetes.io/projected/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-kube-api-access-56l27\") pod \"maas-api-key-cleanup-29894175-zpjj8\" (UID: \"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" Nov 02 20:15:00.267426 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.267409 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894175-zpjj8\" (UID: \"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" Nov 02 20:15:00.267564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.267545 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e758972f-7e9f-4e99-9339-f41ec7c7600d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz\" (UID: \"e758972f-7e9f-4e99-9339-f41ec7c7600d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" Nov 02 20:15:00.267663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.267637 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w22vj\" (UniqueName: \"kubernetes.io/projected/e758972f-7e9f-4e99-9339-f41ec7c7600d-kube-api-access-w22vj\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz\" (UID: \"e758972f-7e9f-4e99-9339-f41ec7c7600d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" Nov 02 20:15:00.269265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.269237 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4-kube-api-access-npdmc" pod "e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" (UID: "e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4"). InnerVolumeSpecName "kube-api-access-npdmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:00.306146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.303064 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822"] Nov 02 20:15:00.369347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.369240 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56l27\" (UniqueName: \"kubernetes.io/projected/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-kube-api-access-56l27\") pod \"maas-api-key-cleanup-29894175-zpjj8\" (UID: \"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" Nov 02 20:15:00.369511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.369403 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894175-zpjj8\" (UID: \"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" Nov 02 20:15:00.369589 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.369519 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e758972f-7e9f-4e99-9339-f41ec7c7600d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz\" (UID: \"e758972f-7e9f-4e99-9339-f41ec7c7600d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" Nov 02 20:15:00.369589 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.369559 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w22vj\" (UniqueName: \"kubernetes.io/projected/e758972f-7e9f-4e99-9339-f41ec7c7600d-kube-api-access-w22vj\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz\" (UID: \"e758972f-7e9f-4e99-9339-f41ec7c7600d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" Nov 02 20:15:00.369682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.369651 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02079f5-208b-4062-8861-b4088cc6b3c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822\" (UID: \"f02079f5-208b-4062-8861-b4088cc6b3c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" Nov 02 20:15:00.369737 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.369706 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4pks\" (UniqueName: \"kubernetes.io/projected/f02079f5-208b-4062-8861-b4088cc6b3c3-kube-api-access-k4pks\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822\" (UID: \"f02079f5-208b-4062-8861-b4088cc6b3c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" Nov 02 20:15:00.369960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.369934 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npdmc\" (UniqueName: \"kubernetes.io/projected/e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4-kube-api-access-npdmc\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:00.370183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.370164 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e758972f-7e9f-4e99-9339-f41ec7c7600d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz\" (UID: \"e758972f-7e9f-4e99-9339-f41ec7c7600d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" Nov 02 20:15:00.370226 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.370179 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894175-zpjj8\" (UID: \"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" Nov 02 20:15:00.422155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.422126 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56l27\" (UniqueName: \"kubernetes.io/projected/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-kube-api-access-56l27\") pod \"maas-api-key-cleanup-29894175-zpjj8\" (UID: \"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" Nov 02 20:15:00.422366 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.422347 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w22vj\" (UniqueName: \"kubernetes.io/projected/e758972f-7e9f-4e99-9339-f41ec7c7600d-kube-api-access-w22vj\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz\" (UID: \"e758972f-7e9f-4e99-9339-f41ec7c7600d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" Nov 02 20:15:00.433085 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.433050 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7198-66c4c1"] Nov 02 20:15:00.438909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.438875 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7198-66c4c1"] Nov 02 20:15:00.470794 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:00.470739 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-f080ff\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:15:00.473204 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.473172 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02079f5-208b-4062-8861-b4088cc6b3c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822\" (UID: \"f02079f5-208b-4062-8861-b4088cc6b3c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" Nov 02 20:15:00.473437 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.473238 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02079f5-208b-4062-8861-b4088cc6b3c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822\" (UID: \"f02079f5-208b-4062-8861-b4088cc6b3c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" Nov 02 20:15:00.473437 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.473325 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4pks\" (UniqueName: \"kubernetes.io/projected/f02079f5-208b-4062-8861-b4088cc6b3c3-kube-api-access-k4pks\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822\" (UID: \"f02079f5-208b-4062-8861-b4088cc6b3c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" Nov 02 20:15:00.483964 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.482979 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" Nov 02 20:15:00.490018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.489991 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" Nov 02 20:15:00.528713 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.528674 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4pks\" (UniqueName: \"kubernetes.io/projected/f02079f5-208b-4062-8861-b4088cc6b3c3-kube-api-access-k4pks\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822\" (UID: \"f02079f5-208b-4062-8861-b4088cc6b3c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" Nov 02 20:15:00.818237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.818197 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" Nov 02 20:15:00.871448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.871141 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz"] Nov 02 20:15:00.873342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.872088 5399 generic.go:356] "Generic (PLEG): container finished" podID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" containerID="324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79" exitCode=137 Nov 02 20:15:00.873342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.872146 5399 scope.go:122] "RemoveContainer" containerID="324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79" Nov 02 20:15:00.873342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.872150 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:15:00.873342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.872499 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7198-66c4c1" podUID="2a028cdd-b2a9-4751-9c35-a916d21d1b10" containerName="test-curl-7198-66c4c1" containerID="cri-o://a7a6acf5478c028ee68ced84403e5bd72b52b5caf2feb81e278f264a58c931ee" gracePeriod=2 Nov 02 20:15:00.875462 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:15:00.874569 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode758972f_7e9f_4e99_9339_f41ec7c7600d.slice/crio-d8426ae065fc380ce9ae91cc9d122d60ddfe5532168591cbd60f50f66108a2fd WatchSource:0}: Error finding container d8426ae065fc380ce9ae91cc9d122d60ddfe5532168591cbd60f50f66108a2fd: Status 404 returned error can't find the container with id d8426ae065fc380ce9ae91cc9d122d60ddfe5532168591cbd60f50f66108a2fd Nov 02 20:15:00.905218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.905114 5399 scope.go:122] "RemoveContainer" containerID="324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79" Nov 02 20:15:00.905588 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:00.905507 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79\": container with ID starting with 324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79 not found: ID does not exist" containerID="324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79" Nov 02 20:15:00.905588 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.905546 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79"} err="failed to get container status \"324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79\": rpc error: code = NotFound desc = could not find container \"324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79\": container with ID starting with 324f5d3155e59534b90912acaf7ef249f2fe478245b07bd4a9e1a4e208235b79 not found: ID does not exist" Nov 02 20:15:00.924011 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:00.923988 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8"] Nov 02 20:15:00.926701 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:00.926590 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-f080ff\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:15:00.974324 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:00.974081 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-66c4c1\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="2a028cdd-b2a9-4751-9c35-a916d21d1b10" pod="opendatahub/test-curl-7198-66c4c1" Nov 02 20:15:01.021630 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:01.021595 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-f080ff\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" pod="opendatahub/test-curl-7198-f080ff" Nov 02 20:15:01.069175 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:01.069129 5399 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-66c4c1\" is forbidden: User \"system:node:ip-10-0-130-111.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-111.us-west-2.compute.internal' and this object" podUID="2a028cdd-b2a9-4751-9c35-a916d21d1b10" pod="opendatahub/test-curl-7198-66c4c1" Nov 02 20:15:01.212877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:01.212771 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822"] Nov 02 20:15:01.213521 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:15:01.213486 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02079f5_208b_4062_8861_b4088cc6b3c3.slice/crio-8e276b5415d0962ad0bd895ade5d39fd09ff5929c0544d869028ffdfed92e471 WatchSource:0}: Error finding container 8e276b5415d0962ad0bd895ade5d39fd09ff5929c0544d869028ffdfed92e471: Status 404 returned error can't find the container with id 8e276b5415d0962ad0bd895ade5d39fd09ff5929c0544d869028ffdfed92e471 Nov 02 20:15:01.533355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:01.531019 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4" path="/var/lib/kubelet/pods/e5c6faf9-b9be-4069-9d3b-1440d1d1b2f4/volumes" Nov 02 20:15:01.886453 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:01.886409 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" event={"ID":"f02079f5-208b-4062-8861-b4088cc6b3c3","Type":"ContainerStarted","Data":"b1320803e1dae6487be93c3589e6714cd34831f52b390ef7c9f4b0e8ee13bab0"} Nov 02 20:15:01.887362 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:01.887024 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" event={"ID":"f02079f5-208b-4062-8861-b4088cc6b3c3","Type":"ContainerStarted","Data":"8e276b5415d0962ad0bd895ade5d39fd09ff5929c0544d869028ffdfed92e471"} Nov 02 20:15:01.888744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:01.888586 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" event={"ID":"e758972f-7e9f-4e99-9339-f41ec7c7600d","Type":"ContainerStarted","Data":"1af92e6c95cee98d6f9c6587b6e20561ec9586f21339cdbb8c57d1e495cbb5bc"} Nov 02 20:15:01.888744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:01.888636 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" event={"ID":"e758972f-7e9f-4e99-9339-f41ec7c7600d","Type":"ContainerStarted","Data":"d8426ae065fc380ce9ae91cc9d122d60ddfe5532168591cbd60f50f66108a2fd"} Nov 02 20:15:01.891688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:01.891656 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" event={"ID":"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1","Type":"ContainerStarted","Data":"ce0e844c205ac3a27061625626336fd30ab5a9d229d1cd1a850f5fe201ffd982"} Nov 02 20:15:01.891802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:01.891707 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" event={"ID":"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1","Type":"ContainerStarted","Data":"47f82f862861b0d9c41677841e56dc2df79168dd615d58b9fa44c4cc40a20c9a"} Nov 02 20:15:01.995913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:01.995814 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" podStartSLOduration=1.995797206 podStartE2EDuration="1.995797206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:01.99222704 +0000 UTC m=+1344.602879132" watchObservedRunningTime="2026-11-02 20:15:01.995797206 +0000 UTC m=+1344.606449364" Nov 02 20:15:02.110978 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.110901 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" podStartSLOduration=2.110885204 podStartE2EDuration="2.110885204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.10902755 +0000 UTC m=+1344.719679642" watchObservedRunningTime="2026-11-02 20:15:02.110885204 +0000 UTC m=+1344.721537295" Nov 02 20:15:02.242738 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.242683 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" podStartSLOduration=2.242665708 podStartE2EDuration="2.242665708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:02.239435058 +0000 UTC m=+1344.850087150" watchObservedRunningTime="2026-11-02 20:15:02.242665708 +0000 UTC m=+1344.853317850" Nov 02 20:15:02.382881 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.382843 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs"] Nov 02 20:15:02.383530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.383515 5399 cpu_manager.go:414] "containerMap: removing container" podUID="2a028cdd-b2a9-4751-9c35-a916d21d1b10" containerName="test-curl-7198-66c4c1" Nov 02 20:15:02.383579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.383531 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a028cdd-b2a9-4751-9c35-a916d21d1b10" containerName="test-curl-7198-66c4c1" Nov 02 20:15:02.383679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.383669 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a028cdd-b2a9-4751-9c35-a916d21d1b10" containerName="test-curl-7198-66c4c1" Nov 02 20:15:02.387354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.387334 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.425423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.425369 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-certs\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.425599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.425472 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-credential-socket\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.425599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.425506 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-envoy\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.425599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.425577 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d83953ff-1dd7-4b12-bc48-90ef3658777d-istiod-ca-cert\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.425791 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.425775 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-data\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.425865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.425826 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-podinfo\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.426061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.426040 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-token\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.426194 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.426086 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-socket\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.426194 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.426152 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbr2b\" (UniqueName: \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-kube-api-access-vbr2b\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.440882 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.440848 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2ac9784d-gw-openshift-default-dockercfg-jzhlb\"" Nov 02 20:15:02.443841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.443803 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs"] Nov 02 20:15:02.527760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.527722 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-data\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.527966 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.527823 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-podinfo\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.527966 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.527873 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-token\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.528105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.527973 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-socket\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.528105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.528029 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbr2b\" (UniqueName: \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-kube-api-access-vbr2b\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.528105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.528079 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-certs\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.528263 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.528115 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-credential-socket\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.528263 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.528153 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-envoy\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.528263 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.528254 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-data\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.528456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.528324 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d83953ff-1dd7-4b12-bc48-90ef3658777d-istiod-ca-cert\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.528528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.528473 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-socket\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.528946 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.528916 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-credential-socket\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.529114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.529092 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-certs\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.529337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.529317 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d83953ff-1dd7-4b12-bc48-90ef3658777d-istiod-ca-cert\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.530744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.530723 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-envoy\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.531044 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.531024 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-podinfo\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.581291 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.581235 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-token\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.581448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.581368 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbr2b\" (UniqueName: \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-kube-api-access-vbr2b\") pod \"e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.703480 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.703376 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:02.909969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.909923 5399 generic.go:356] "Generic (PLEG): container finished" podID="2a028cdd-b2a9-4751-9c35-a916d21d1b10" containerID="a7a6acf5478c028ee68ced84403e5bd72b52b5caf2feb81e278f264a58c931ee" exitCode=137 Nov 02 20:15:02.912238 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.912208 5399 generic.go:356] "Generic (PLEG): container finished" podID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerID="b1320803e1dae6487be93c3589e6714cd34831f52b390ef7c9f4b0e8ee13bab0" exitCode=60 Nov 02 20:15:02.912399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.912316 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" event={"ID":"f02079f5-208b-4062-8861-b4088cc6b3c3","Type":"ContainerDied","Data":"b1320803e1dae6487be93c3589e6714cd34831f52b390ef7c9f4b0e8ee13bab0"} Nov 02 20:15:02.912691 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.912676 5399 scope.go:122] "RemoveContainer" containerID="b1320803e1dae6487be93c3589e6714cd34831f52b390ef7c9f4b0e8ee13bab0" Nov 02 20:15:02.914407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.914323 5399 generic.go:356] "Generic (PLEG): container finished" podID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerID="1af92e6c95cee98d6f9c6587b6e20561ec9586f21339cdbb8c57d1e495cbb5bc" exitCode=60 Nov 02 20:15:02.914407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.914361 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" event={"ID":"e758972f-7e9f-4e99-9339-f41ec7c7600d","Type":"ContainerDied","Data":"1af92e6c95cee98d6f9c6587b6e20561ec9586f21339cdbb8c57d1e495cbb5bc"} Nov 02 20:15:02.914743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.914725 5399 scope.go:122] "RemoveContainer" containerID="1af92e6c95cee98d6f9c6587b6e20561ec9586f21339cdbb8c57d1e495cbb5bc" Nov 02 20:15:02.916800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.916382 5399 generic.go:356] "Generic (PLEG): container finished" podID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerID="ce0e844c205ac3a27061625626336fd30ab5a9d229d1cd1a850f5fe201ffd982" exitCode=60 Nov 02 20:15:02.916800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.916440 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" event={"ID":"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1","Type":"ContainerDied","Data":"ce0e844c205ac3a27061625626336fd30ab5a9d229d1cd1a850f5fe201ffd982"} Nov 02 20:15:02.916800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:02.916710 5399 scope.go:122] "RemoveContainer" containerID="ce0e844c205ac3a27061625626336fd30ab5a9d229d1cd1a850f5fe201ffd982" Nov 02 20:15:03.139993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.139964 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-66c4c1" Nov 02 20:15:03.175450 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.175420 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs"] Nov 02 20:15:03.176107 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:15:03.176077 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83953ff_1dd7_4b12_bc48_90ef3658777d.slice/crio-687a0cdcd23f12859660cfc1c83833cea46e69c340d4337a2a5fc0f239f12b7a WatchSource:0}: Error finding container 687a0cdcd23f12859660cfc1c83833cea46e69c340d4337a2a5fc0f239f12b7a: Status 404 returned error can't find the container with id 687a0cdcd23f12859660cfc1c83833cea46e69c340d4337a2a5fc0f239f12b7a Nov 02 20:15:03.178813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.178781 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:15:03.178898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.178842 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:15:03.178898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.178874 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:15:03.253210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.253169 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a028cdd-b2a9-4751-9c35-a916d21d1b10-kube-api-access-qzwq2\" (UniqueName: \"kubernetes.io/projected/2a028cdd-b2a9-4751-9c35-a916d21d1b10-kube-api-access-qzwq2\") pod \"2a028cdd-b2a9-4751-9c35-a916d21d1b10\" (UID: \"2a028cdd-b2a9-4751-9c35-a916d21d1b10\") " Nov 02 20:15:03.327240 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.327193 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a028cdd-b2a9-4751-9c35-a916d21d1b10-kube-api-access-qzwq2" pod "2a028cdd-b2a9-4751-9c35-a916d21d1b10" (UID: "2a028cdd-b2a9-4751-9c35-a916d21d1b10"). InnerVolumeSpecName "kube-api-access-qzwq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:03.355461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.355433 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwq2\" (UniqueName: \"kubernetes.io/projected/2a028cdd-b2a9-4751-9c35-a916d21d1b10-kube-api-access-qzwq2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:03.509747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.509667 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a028cdd-b2a9-4751-9c35-a916d21d1b10" path="/var/lib/kubelet/pods/2a028cdd-b2a9-4751-9c35-a916d21d1b10/volumes" Nov 02 20:15:03.926719 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.926684 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" event={"ID":"f02079f5-208b-4062-8861-b4088cc6b3c3","Type":"ContainerStarted","Data":"f6165ff52b46cdcc2d3b02be43cb8ad6fe2e9164aa91570a0ae2d4d8396f1486"} Nov 02 20:15:03.929334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.929300 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" event={"ID":"e758972f-7e9f-4e99-9339-f41ec7c7600d","Type":"ContainerStarted","Data":"5cf9f40c732f833a547b59ecfcca7d96a4e813e2b1effb3e036acaeb48fadd54"} Nov 02 20:15:03.931509 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.931484 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" event={"ID":"d83953ff-1dd7-4b12-bc48-90ef3658777d","Type":"ContainerStarted","Data":"56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92"} Nov 02 20:15:03.931627 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.931535 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" event={"ID":"d83953ff-1dd7-4b12-bc48-90ef3658777d","Type":"ContainerStarted","Data":"687a0cdcd23f12859660cfc1c83833cea46e69c340d4337a2a5fc0f239f12b7a"} Nov 02 20:15:03.934134 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.933684 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" event={"ID":"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1","Type":"ContainerStarted","Data":"6020e47b5688f7b2f4122a0c255154656d8205edfb8f8d196c816d6d318b44d2"} Nov 02 20:15:03.935803 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.935772 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-66c4c1" Nov 02 20:15:03.935803 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:03.935787 5399 scope.go:122] "RemoveContainer" containerID="a7a6acf5478c028ee68ced84403e5bd72b52b5caf2feb81e278f264a58c931ee" Nov 02 20:15:04.352495 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.352436 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" podStartSLOduration=2.3524201319999998 podStartE2EDuration="2.352420132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:04.3511345 +0000 UTC m=+1346.961786592" watchObservedRunningTime="2026-11-02 20:15:04.352420132 +0000 UTC m=+1346.963072219" Nov 02 20:15:04.703562 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.703529 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:04.708936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.708911 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:04.944647 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.944617 5399 generic.go:356] "Generic (PLEG): container finished" podID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerID="f6165ff52b46cdcc2d3b02be43cb8ad6fe2e9164aa91570a0ae2d4d8396f1486" exitCode=60 Nov 02 20:15:04.944815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.944698 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" event={"ID":"f02079f5-208b-4062-8861-b4088cc6b3c3","Type":"ContainerDied","Data":"f6165ff52b46cdcc2d3b02be43cb8ad6fe2e9164aa91570a0ae2d4d8396f1486"} Nov 02 20:15:04.944815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.944764 5399 scope.go:122] "RemoveContainer" containerID="b1320803e1dae6487be93c3589e6714cd34831f52b390ef7c9f4b0e8ee13bab0" Nov 02 20:15:04.945049 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.945036 5399 scope.go:122] "RemoveContainer" containerID="f6165ff52b46cdcc2d3b02be43cb8ad6fe2e9164aa91570a0ae2d4d8396f1486" Nov 02 20:15:04.945466 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:04.945439 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822_odh-ai-gateway-infra(f02079f5-208b-4062-8861-b4088cc6b3c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" Nov 02 20:15:04.946936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.946908 5399 generic.go:356] "Generic (PLEG): container finished" podID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerID="5cf9f40c732f833a547b59ecfcca7d96a4e813e2b1effb3e036acaeb48fadd54" exitCode=60 Nov 02 20:15:04.947034 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.946986 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" event={"ID":"e758972f-7e9f-4e99-9339-f41ec7c7600d","Type":"ContainerDied","Data":"5cf9f40c732f833a547b59ecfcca7d96a4e813e2b1effb3e036acaeb48fadd54"} Nov 02 20:15:04.947253 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.947236 5399 scope.go:122] "RemoveContainer" containerID="5cf9f40c732f833a547b59ecfcca7d96a4e813e2b1effb3e036acaeb48fadd54" Nov 02 20:15:04.947632 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:04.947609 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz_odh-ai-gateway-infra(e758972f-7e9f-4e99-9339-f41ec7c7600d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" Nov 02 20:15:04.948995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.948974 5399 generic.go:356] "Generic (PLEG): container finished" podID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerID="6020e47b5688f7b2f4122a0c255154656d8205edfb8f8d196c816d6d318b44d2" exitCode=60 Nov 02 20:15:04.949077 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.949053 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" event={"ID":"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1","Type":"ContainerDied","Data":"6020e47b5688f7b2f4122a0c255154656d8205edfb8f8d196c816d6d318b44d2"} Nov 02 20:15:04.949394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.949378 5399 scope.go:122] "RemoveContainer" containerID="6020e47b5688f7b2f4122a0c255154656d8205edfb8f8d196c816d6d318b44d2" Nov 02 20:15:04.949702 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:04.949681 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894175-zpjj8_odh-ai-gateway-infra(251b41b6-5a95-4e5e-a36d-c2b2ab56cac1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" Nov 02 20:15:04.950567 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.950547 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:04.951643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.951620 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:15:04.961595 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.961572 5399 scope.go:122] "RemoveContainer" containerID="1af92e6c95cee98d6f9c6587b6e20561ec9586f21339cdbb8c57d1e495cbb5bc" Nov 02 20:15:04.976162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:04.976137 5399 scope.go:122] "RemoveContainer" containerID="ce0e844c205ac3a27061625626336fd30ab5a9d229d1cd1a850f5fe201ffd982" Nov 02 20:15:05.438887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.438845 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h"] Nov 02 20:15:05.451185 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.451143 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.476941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.476904 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h"] Nov 02 20:15:05.499445 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.499407 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdzxm\" (UniqueName: \"kubernetes.io/projected/76ce1fd1-4f23-4b6b-9242-91ff525fae58-kube-api-access-vdzxm\") pod \"maas-api-revoke-sub-e003b6ec-ggr7h\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.499619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.499550 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:15:05.499681 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.499628 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e003b6ec-ggr7h\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.499788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.499768 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e003b6ec-ggr7h\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.601462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.601427 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e003b6ec-ggr7h\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.601679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.601565 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdzxm\" (UniqueName: \"kubernetes.io/projected/76ce1fd1-4f23-4b6b-9242-91ff525fae58-kube-api-access-vdzxm\") pod \"maas-api-revoke-sub-e003b6ec-ggr7h\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.601679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.601665 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e003b6ec-ggr7h\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.602059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.602034 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e003b6ec-ggr7h\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.604094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.604070 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e003b6ec-ggr7h\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.654809 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.654782 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdzxm\" (UniqueName: \"kubernetes.io/projected/76ce1fd1-4f23-4b6b-9242-91ff525fae58-kube-api-access-vdzxm\") pod \"maas-api-revoke-sub-e003b6ec-ggr7h\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.767212 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.767138 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:05.959718 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.959689 5399 scope.go:122] "RemoveContainer" containerID="f6165ff52b46cdcc2d3b02be43cb8ad6fe2e9164aa91570a0ae2d4d8396f1486" Nov 02 20:15:05.960127 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:05.960100 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822_odh-ai-gateway-infra(f02079f5-208b-4062-8861-b4088cc6b3c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" Nov 02 20:15:05.961510 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.961486 5399 scope.go:122] "RemoveContainer" containerID="5cf9f40c732f833a547b59ecfcca7d96a4e813e2b1effb3e036acaeb48fadd54" Nov 02 20:15:05.961838 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:05.961816 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz_odh-ai-gateway-infra(e758972f-7e9f-4e99-9339-f41ec7c7600d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" Nov 02 20:15:05.963334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:05.963314 5399 scope.go:122] "RemoveContainer" containerID="6020e47b5688f7b2f4122a0c255154656d8205edfb8f8d196c816d6d318b44d2" Nov 02 20:15:05.963619 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:05.963598 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894175-zpjj8_odh-ai-gateway-infra(251b41b6-5a95-4e5e-a36d-c2b2ab56cac1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" Nov 02 20:15:06.176787 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:06.176757 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h"] Nov 02 20:15:06.177187 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:15:06.177169 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ce1fd1_4f23_4b6b_9242_91ff525fae58.slice/crio-968f6fc48a7439141fdfd062e8f0bf52606727ddbdb1ca7cb8ba27f4db98470c WatchSource:0}: Error finding container 968f6fc48a7439141fdfd062e8f0bf52606727ddbdb1ca7cb8ba27f4db98470c: Status 404 returned error can't find the container with id 968f6fc48a7439141fdfd062e8f0bf52606727ddbdb1ca7cb8ba27f4db98470c Nov 02 20:15:06.972801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:06.972763 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" event={"ID":"76ce1fd1-4f23-4b6b-9242-91ff525fae58","Type":"ContainerStarted","Data":"d6051846cec41b56589cefe069a7fc4bd551aee8b04a99d00f92d011e7712fd1"} Nov 02 20:15:06.973321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:06.972821 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" event={"ID":"76ce1fd1-4f23-4b6b-9242-91ff525fae58","Type":"ContainerStarted","Data":"968f6fc48a7439141fdfd062e8f0bf52606727ddbdb1ca7cb8ba27f4db98470c"} Nov 02 20:15:07.078605 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:07.078550 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" podStartSLOduration=2.078533167 podStartE2EDuration="2.078533167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:07.077376186 +0000 UTC m=+1349.688028274" watchObservedRunningTime="2026-11-02 20:15:07.078533167 +0000 UTC m=+1349.689185258" Nov 02 20:15:07.981813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:07.981776 5399 generic.go:356] "Generic (PLEG): container finished" podID="76ce1fd1-4f23-4b6b-9242-91ff525fae58" containerID="d6051846cec41b56589cefe069a7fc4bd551aee8b04a99d00f92d011e7712fd1" exitCode=0 Nov 02 20:15:07.982195 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:07.981864 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" event={"ID":"76ce1fd1-4f23-4b6b-9242-91ff525fae58","Type":"ContainerDied","Data":"d6051846cec41b56589cefe069a7fc4bd551aee8b04a99d00f92d011e7712fd1"} Nov 02 20:15:10.430484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.430380 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.542493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.542447 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.642301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.596378 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz"] Nov 02 20:15:10.702333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.697761 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.702333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.701059 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:15:10.748877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.748406 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz"] Nov 02 20:15:10.751065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.751026 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2ac9784d\"" Nov 02 20:15:10.751470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.751450 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2ac9784d-dockercfg-r8ptv\"" Nov 02 20:15:10.755305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.753742 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:10.768224 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.768178 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.887771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.887728 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-xks-ca\") pod \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " Nov 02 20:15:10.888394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.887679 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.896320 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.888635 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-openshift-ca\") pod \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " Nov 02 20:15:10.896489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.896402 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ce1fd1-4f23-4b6b-9242-91ff525fae58-kube-api-access-vdzxm\" (UniqueName: \"kubernetes.io/projected/76ce1fd1-4f23-4b6b-9242-91ff525fae58-kube-api-access-vdzxm\") pod \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\" (UID: \"76ce1fd1-4f23-4b6b-9242-91ff525fae58\") " Nov 02 20:15:10.896622 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.895508 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-openshift-ca" pod "76ce1fd1-4f23-4b6b-9242-91ff525fae58" (UID: "76ce1fd1-4f23-4b6b-9242-91ff525fae58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:10.897056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.896879 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4g87\" (UniqueName: \"kubernetes.io/projected/7c044378-c69b-45cb-893c-0d98abf6cb85-kube-api-access-h4g87\") pod \"payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz\" (UID: \"7c044378-c69b-45cb-893c-0d98abf6cb85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:15:10.897056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.896948 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c044378-c69b-45cb-893c-0d98abf6cb85-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz\" (UID: \"7c044378-c69b-45cb-893c-0d98abf6cb85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:15:10.903911 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.903887 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29"] Nov 02 20:15:10.916791 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.916757 5399 cpu_manager.go:414] "containerMap: removing container" podUID="76ce1fd1-4f23-4b6b-9242-91ff525fae58" containerName="revoke-keys" Nov 02 20:15:10.917048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.917034 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ce1fd1-4f23-4b6b-9242-91ff525fae58" containerName="revoke-keys" Nov 02 20:15:10.921329 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.921306 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ce1fd1-4f23-4b6b-9242-91ff525fae58" containerName="revoke-keys" Nov 02 20:15:10.938499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.937454 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:10.966642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.963423 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 02 20:15:10.967735 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.967700 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.969334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.969307 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.969784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.969759 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 20:15:10.970760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.970728 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:10.996587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.983338 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:15:10.996587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.988065 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29"] Nov 02 20:15:10.996587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.988598 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:15:11.006883 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:10.997582 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.136302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.054791 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njt5h\" (UniqueName: \"kubernetes.io/projected/c894a85f-10e2-44d9-993b-c50b43ad9734-kube-api-access-njt5h\") pod \"payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29\" (UID: \"c894a85f-10e2-44d9-993b-c50b43ad9734\") " pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:15:11.144299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.137127 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c894a85f-10e2-44d9-993b-c50b43ad9734-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29\" (UID: \"c894a85f-10e2-44d9-993b-c50b43ad9734\") " pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:15:11.174302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.144673 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4g87\" (UniqueName: \"kubernetes.io/projected/7c044378-c69b-45cb-893c-0d98abf6cb85-kube-api-access-h4g87\") pod \"payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz\" (UID: \"7c044378-c69b-45cb-893c-0d98abf6cb85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:15:11.175088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.175054 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c044378-c69b-45cb-893c-0d98abf6cb85-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz\" (UID: \"7c044378-c69b-45cb-893c-0d98abf6cb85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:15:11.176553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.176533 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c044378-c69b-45cb-893c-0d98abf6cb85-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz\" (UID: \"7c044378-c69b-45cb-893c-0d98abf6cb85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:15:11.176710 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.096985 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-xks-ca" pod "76ce1fd1-4f23-4b6b-9242-91ff525fae58" (UID: "76ce1fd1-4f23-4b6b-9242-91ff525fae58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:15:11.299958 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.299917 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njt5h\" (UniqueName: \"kubernetes.io/projected/c894a85f-10e2-44d9-993b-c50b43ad9734-kube-api-access-njt5h\") pod \"payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29\" (UID: \"c894a85f-10e2-44d9-993b-c50b43ad9734\") " pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:15:11.300555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.300513 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c894a85f-10e2-44d9-993b-c50b43ad9734-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29\" (UID: \"c894a85f-10e2-44d9-993b-c50b43ad9734\") " pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:15:11.300845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.300821 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ce1fd1-4f23-4b6b-9242-91ff525fae58-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:11.339696 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.339662 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c894a85f-10e2-44d9-993b-c50b43ad9734-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29\" (UID: \"c894a85f-10e2-44d9-993b-c50b43ad9734\") " pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:15:11.497768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.497722 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4g87\" (UniqueName: \"kubernetes.io/projected/7c044378-c69b-45cb-893c-0d98abf6cb85-kube-api-access-h4g87\") pod \"payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz\" (UID: \"7c044378-c69b-45cb-893c-0d98abf6cb85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:15:11.615377 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.527510 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:15:11.615377 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.608825 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.673300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.672385 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:15:11.673300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.672467 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.803256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.803207 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ce1fd1-4f23-4b6b-9242-91ff525fae58-kube-api-access-vdzxm" pod "76ce1fd1-4f23-4b6b-9242-91ff525fae58" (UID: "76ce1fd1-4f23-4b6b-9242-91ff525fae58"). InnerVolumeSpecName "kube-api-access-vdzxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:11.807624 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.804147 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njt5h\" (UniqueName: \"kubernetes.io/projected/c894a85f-10e2-44d9-993b-c50b43ad9734-kube-api-access-njt5h\") pod \"payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29\" (UID: \"c894a85f-10e2-44d9-993b-c50b43ad9734\") " pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:15:11.826303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.821994 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:15:11.867205 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.828636 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdzxm\" (UniqueName: \"kubernetes.io/projected/76ce1fd1-4f23-4b6b-9242-91ff525fae58-kube-api-access-vdzxm\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:11.898415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.898368 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:11.917083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.917030 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:15:11.952248 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.950774 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:15:11.988299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.962718 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" event={"ID":"76ce1fd1-4f23-4b6b-9242-91ff525fae58","Type":"ContainerDied","Data":"968f6fc48a7439141fdfd062e8f0bf52606727ddbdb1ca7cb8ba27f4db98470c"} Nov 02 20:15:11.988607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.988575 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="968f6fc48a7439141fdfd062e8f0bf52606727ddbdb1ca7cb8ba27f4db98470c" Nov 02 20:15:12.017351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.965412 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.017951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.975061 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.050301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:12.039513 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:15:12.050301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:12.039608 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:15:12.050301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.975139 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.050301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.976043 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:15:12.050301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.976076 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:15:12.050301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:12.009391 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 20:15:12.050750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:12.050516 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:15:12.062315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:12.054343 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:15:12.062315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:12.054432 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:15:12.062315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:12.054486 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:15:12.088564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:12.009427 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:12.088564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:12.081495 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:15:12.088564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:11.964446 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h" Nov 02 20:15:12.987923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:12.987878 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.202452 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.166584 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.328374 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.292836 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.413302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.387751 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.780357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.765919 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-d5kgg" podUID="d2f7e6ae-694c-43dc-b392-a6414481c8cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.786390 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.786022 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mkqnr" podUID="930a980d-dfca-4c6e-964f-38a7496267fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Nov 02 20:15:13.849393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.787967 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:13.849393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.788242 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:15:13.879586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.879539 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 02 20:15:13.897230 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.897187 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 02 20:15:14.368268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.552609 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:15:14.411315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.552766 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr" podUID="104f859d-9978-4548-91da-a1d3e1438dca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:14.477690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:13.574446 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-dxf74" podUID="7d6e94b5-d4a7-48d4-ab4b-27b457ed367c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:14.701799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:14.701756 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 20:15:15.172136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.146624 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 02 20:15:15.172136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.163512 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:15:15.172136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.163792 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:15:15.172136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.163983 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Nov 02 20:15:15.172136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.164035 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:15:15.172136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.164219 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 02 20:15:15.172136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.164418 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Nov 02 20:15:15.172136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.164485 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:15.172136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.164645 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:15:15.339292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.275297 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:15:15.339292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.275363 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:15.339292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.275446 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:15:15.339292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.275467 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:15.344770 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:15.344723 5399 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.822s" Nov 02 20:15:15.345115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.345094 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz"] Nov 02 20:15:15.346945 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.346895 5399 scope.go:122] "RemoveContainer" containerID="f6165ff52b46cdcc2d3b02be43cb8ad6fe2e9164aa91570a0ae2d4d8396f1486" Nov 02 20:15:15.405088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.405062 5399 scope.go:122] "RemoveContainer" containerID="6020e47b5688f7b2f4122a0c255154656d8205edfb8f8d196c816d6d318b44d2" Nov 02 20:15:15.444494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.373427 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:15:15.466252 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.466216 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:15:15.466522 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.466509 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:15:15.481269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.481241 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29"] Nov 02 20:15:15.484799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.484780 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:15:15.488593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.488551 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:15:15.492925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.492906 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:15:15.494118 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.493977 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:15:15.542586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.542556 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:15:15.542886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.542623 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:15:15.542886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.542658 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:15:15.898330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.898295 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" event={"ID":"7c044378-c69b-45cb-893c-0d98abf6cb85","Type":"ContainerStarted","Data":"4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44"} Nov 02 20:15:15.898517 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.898359 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" event={"ID":"7c044378-c69b-45cb-893c-0d98abf6cb85","Type":"ContainerStarted","Data":"6d3ceddff156c96daad1e8c2aa31d0fd7238ac64708ff2291770f23fbafe65fb"} Nov 02 20:15:15.898517 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.898407 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:15:15.900080 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.900053 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" event={"ID":"c894a85f-10e2-44d9-993b-c50b43ad9734","Type":"ContainerStarted","Data":"77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b"} Nov 02 20:15:15.900221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.900093 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" event={"ID":"c894a85f-10e2-44d9-993b-c50b43ad9734","Type":"ContainerStarted","Data":"d56c19a31b63e5bde8d8cc03926d42d1b9c91cc9af68b5ce4f3aad344f444cfd"} Nov 02 20:15:15.900306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:15.900233 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:15:16.185837 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.185783 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" podStartSLOduration=6.185768375 podStartE2EDuration="6.185768375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:16.185460592 +0000 UTC m=+1358.796112683" watchObservedRunningTime="2026-11-02 20:15:16.185768375 +0000 UTC m=+1358.796420465" Nov 02 20:15:16.501342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.501247 5399 scope.go:122] "RemoveContainer" containerID="5cf9f40c732f833a547b59ecfcca7d96a4e813e2b1effb3e036acaeb48fadd54" Nov 02 20:15:16.609632 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.609573 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" podStartSLOduration=6.609560098 podStartE2EDuration="6.609560098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:16.605695356 +0000 UTC m=+1359.216347452" watchObservedRunningTime="2026-11-02 20:15:16.609560098 +0000 UTC m=+1359.220212188" Nov 02 20:15:16.909751 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.909678 5399 generic.go:356] "Generic (PLEG): container finished" podID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerID="c59e4d3a57dae15a9a16c42909fa3ecfed4d3711030b6f293e8c2632a6b9e594" exitCode=60 Nov 02 20:15:16.909901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.909736 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" event={"ID":"f02079f5-208b-4062-8861-b4088cc6b3c3","Type":"ContainerDied","Data":"c59e4d3a57dae15a9a16c42909fa3ecfed4d3711030b6f293e8c2632a6b9e594"} Nov 02 20:15:16.909901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.909815 5399 scope.go:122] "RemoveContainer" containerID="f6165ff52b46cdcc2d3b02be43cb8ad6fe2e9164aa91570a0ae2d4d8396f1486" Nov 02 20:15:16.910338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.910311 5399 scope.go:122] "RemoveContainer" containerID="c59e4d3a57dae15a9a16c42909fa3ecfed4d3711030b6f293e8c2632a6b9e594" Nov 02 20:15:16.910762 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:16.910733 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822_odh-ai-gateway-infra(f02079f5-208b-4062-8861-b4088cc6b3c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" Nov 02 20:15:16.911975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.911952 5399 generic.go:356] "Generic (PLEG): container finished" podID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerID="22ddca0427001b8856efa2ad01e8a1dca85a8cdd866d4ee87a2121b6b5cd4261" exitCode=60 Nov 02 20:15:16.912075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.912033 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" event={"ID":"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1","Type":"ContainerDied","Data":"22ddca0427001b8856efa2ad01e8a1dca85a8cdd866d4ee87a2121b6b5cd4261"} Nov 02 20:15:16.912675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.912661 5399 scope.go:122] "RemoveContainer" containerID="22ddca0427001b8856efa2ad01e8a1dca85a8cdd866d4ee87a2121b6b5cd4261" Nov 02 20:15:16.912922 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:16.912904 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894175-zpjj8_odh-ai-gateway-infra(251b41b6-5a95-4e5e-a36d-c2b2ab56cac1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" Nov 02 20:15:16.923887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:16.923867 5399 scope.go:122] "RemoveContainer" containerID="6020e47b5688f7b2f4122a0c255154656d8205edfb8f8d196c816d6d318b44d2" Nov 02 20:15:17.928548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:17.928506 5399 generic.go:356] "Generic (PLEG): container finished" podID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerID="b475a515c8f0eafd3f7413cb802053c19ce13ddd7f7316093a2b131a5b310649" exitCode=60 Nov 02 20:15:17.929052 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:17.928584 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" event={"ID":"e758972f-7e9f-4e99-9339-f41ec7c7600d","Type":"ContainerDied","Data":"b475a515c8f0eafd3f7413cb802053c19ce13ddd7f7316093a2b131a5b310649"} Nov 02 20:15:17.929052 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:17.928632 5399 scope.go:122] "RemoveContainer" containerID="5cf9f40c732f833a547b59ecfcca7d96a4e813e2b1effb3e036acaeb48fadd54" Nov 02 20:15:17.929239 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:17.929062 5399 scope.go:122] "RemoveContainer" containerID="b475a515c8f0eafd3f7413cb802053c19ce13ddd7f7316093a2b131a5b310649" Nov 02 20:15:17.929451 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:17.929426 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz_odh-ai-gateway-infra(e758972f-7e9f-4e99-9339-f41ec7c7600d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" Nov 02 20:15:18.241398 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:18.241255 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:15:18.251885 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.251857 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8"] Nov 02 20:15:18.257408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.257387 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:15:18.307463 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.307427 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a2b4e70c-dockercfg-dldhr\"" Nov 02 20:15:18.307773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.307565 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a2b4e70c\"" Nov 02 20:15:18.312763 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.312728 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8"] Nov 02 20:15:18.354237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.354199 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8"] Nov 02 20:15:18.369489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.369446 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbk2\" (UniqueName: \"kubernetes.io/projected/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-kube-api-access-vnbk2\") pod \"payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8\" (UID: \"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8\") " pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:15:18.369682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.369614 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8\" (UID: \"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8\") " pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:15:18.488846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.488486 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822"] Nov 02 20:15:18.489884 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.489862 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbk2\" (UniqueName: \"kubernetes.io/projected/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-kube-api-access-vnbk2\") pod \"payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8\" (UID: \"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8\") " pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:15:18.490038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.490018 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8\" (UID: \"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8\") " pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:15:18.491036 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.491011 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8\" (UID: \"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8\") " pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:15:18.533227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.533196 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" Nov 02 20:15:18.543430 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.543401 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbk2\" (UniqueName: \"kubernetes.io/projected/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-kube-api-access-vnbk2\") pod \"payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8\" (UID: \"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8\") " pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:15:18.591487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.591445 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-maas-api-service-ca\") pod \"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1\" (UID: \"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1\") " Nov 02 20:15:18.591677 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.591632 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-kube-api-access-56l27\" (UniqueName: \"kubernetes.io/projected/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-kube-api-access-56l27\") pod \"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1\" (UID: \"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1\") " Nov 02 20:15:18.596521 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.595545 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-kube-api-access-56l27" pod "251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" (UID: "251b41b6-5a95-4e5e-a36d-c2b2ab56cac1"). InnerVolumeSpecName "kube-api-access-56l27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.596521 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.596188 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:15:18.613081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.609248 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-maas-api-service-ca" pod "251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" (UID: "251b41b6-5a95-4e5e-a36d-c2b2ab56cac1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.699057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.699018 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.699183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.699060 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56l27\" (UniqueName: \"kubernetes.io/projected/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1-kube-api-access-56l27\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.743309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.742550 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" Nov 02 20:15:18.801412 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.800224 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02079f5-208b-4062-8861-b4088cc6b3c3-kube-api-access-k4pks\" (UniqueName: \"kubernetes.io/projected/f02079f5-208b-4062-8861-b4088cc6b3c3-kube-api-access-k4pks\") pod \"f02079f5-208b-4062-8861-b4088cc6b3c3\" (UID: \"f02079f5-208b-4062-8861-b4088cc6b3c3\") " Nov 02 20:15:18.801412 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.800411 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02079f5-208b-4062-8861-b4088cc6b3c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02079f5-208b-4062-8861-b4088cc6b3c3-maas-api-service-ca\") pod \"f02079f5-208b-4062-8861-b4088cc6b3c3\" (UID: \"f02079f5-208b-4062-8861-b4088cc6b3c3\") " Nov 02 20:15:18.801412 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.800809 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02079f5-208b-4062-8861-b4088cc6b3c3-maas-api-service-ca" pod "f02079f5-208b-4062-8861-b4088cc6b3c3" (UID: "f02079f5-208b-4062-8861-b4088cc6b3c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.801412 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.801058 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f02079f5-208b-4062-8861-b4088cc6b3c3-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.806548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.806518 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02079f5-208b-4062-8861-b4088cc6b3c3-kube-api-access-k4pks" pod "f02079f5-208b-4062-8861-b4088cc6b3c3" (UID: "f02079f5-208b-4062-8861-b4088cc6b3c3"). InnerVolumeSpecName "kube-api-access-k4pks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.902407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.902374 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4pks\" (UniqueName: \"kubernetes.io/projected/f02079f5-208b-4062-8861-b4088cc6b3c3-kube-api-access-k4pks\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.940340 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.940305 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" Nov 02 20:15:18.940839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.940324 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822" event={"ID":"f02079f5-208b-4062-8861-b4088cc6b3c3","Type":"ContainerDied","Data":"8e276b5415d0962ad0bd895ade5d39fd09ff5929c0544d869028ffdfed92e471"} Nov 02 20:15:18.940839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.940402 5399 scope.go:122] "RemoveContainer" containerID="c59e4d3a57dae15a9a16c42909fa3ecfed4d3711030b6f293e8c2632a6b9e594" Nov 02 20:15:18.947053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.947013 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" event={"ID":"251b41b6-5a95-4e5e-a36d-c2b2ab56cac1","Type":"ContainerDied","Data":"47f82f862861b0d9c41677841e56dc2df79168dd615d58b9fa44c4cc40a20c9a"} Nov 02 20:15:18.947184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.947128 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8" Nov 02 20:15:18.955933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:18.955891 5399 scope.go:122] "RemoveContainer" containerID="22ddca0427001b8856efa2ad01e8a1dca85a8cdd866d4ee87a2121b6b5cd4261" Nov 02 20:15:19.041106 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:15:19.041065 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccd1cf9_eaa9_4f2c_b018_8346316cdfb8.slice/crio-78291b308f2169e75f6a6745cbc2a061c7372e486a425409e747b90beb154db8 WatchSource:0}: Error finding container 78291b308f2169e75f6a6745cbc2a061c7372e486a425409e747b90beb154db8: Status 404 returned error can't find the container with id 78291b308f2169e75f6a6745cbc2a061c7372e486a425409e747b90beb154db8 Nov 02 20:15:19.041946 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.041920 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8"] Nov 02 20:15:19.049165 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.049141 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz"] Nov 02 20:15:19.096723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.096694 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822"] Nov 02 20:15:19.104328 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.104254 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc64d3-29894175-77822"] Nov 02 20:15:19.209082 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.209055 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8"] Nov 02 20:15:19.214099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.214073 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894175-zpjj8"] Nov 02 20:15:19.217749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.217723 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" Nov 02 20:15:19.311392 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.311306 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e758972f-7e9f-4e99-9339-f41ec7c7600d-kube-api-access-w22vj\" (UniqueName: \"kubernetes.io/projected/e758972f-7e9f-4e99-9339-f41ec7c7600d-kube-api-access-w22vj\") pod \"e758972f-7e9f-4e99-9339-f41ec7c7600d\" (UID: \"e758972f-7e9f-4e99-9339-f41ec7c7600d\") " Nov 02 20:15:19.311564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.311429 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e758972f-7e9f-4e99-9339-f41ec7c7600d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e758972f-7e9f-4e99-9339-f41ec7c7600d-maas-api-service-ca\") pod \"e758972f-7e9f-4e99-9339-f41ec7c7600d\" (UID: \"e758972f-7e9f-4e99-9339-f41ec7c7600d\") " Nov 02 20:15:19.311796 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.311773 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e758972f-7e9f-4e99-9339-f41ec7c7600d-maas-api-service-ca" pod "e758972f-7e9f-4e99-9339-f41ec7c7600d" (UID: "e758972f-7e9f-4e99-9339-f41ec7c7600d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:19.311893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.311880 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e758972f-7e9f-4e99-9339-f41ec7c7600d-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.313758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.313731 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e758972f-7e9f-4e99-9339-f41ec7c7600d-kube-api-access-w22vj" pod "e758972f-7e9f-4e99-9339-f41ec7c7600d" (UID: "e758972f-7e9f-4e99-9339-f41ec7c7600d"). InnerVolumeSpecName "kube-api-access-w22vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:19.413918 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.413881 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w22vj\" (UniqueName: \"kubernetes.io/projected/e758972f-7e9f-4e99-9339-f41ec7c7600d-kube-api-access-w22vj\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:19.510433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.510392 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" path="/var/lib/kubelet/pods/251b41b6-5a95-4e5e-a36d-c2b2ab56cac1/volumes" Nov 02 20:15:19.511033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.511008 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" path="/var/lib/kubelet/pods/f02079f5-208b-4062-8861-b4088cc6b3c3/volumes" Nov 02 20:15:19.958002 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.957922 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" Nov 02 20:15:19.958452 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.957928 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz" event={"ID":"e758972f-7e9f-4e99-9339-f41ec7c7600d","Type":"ContainerDied","Data":"d8426ae065fc380ce9ae91cc9d122d60ddfe5532168591cbd60f50f66108a2fd"} Nov 02 20:15:19.958452 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.958070 5399 scope.go:122] "RemoveContainer" containerID="b475a515c8f0eafd3f7413cb802053c19ce13ddd7f7316093a2b131a5b310649" Nov 02 20:15:19.961440 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.961405 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" event={"ID":"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8","Type":"ContainerStarted","Data":"d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001"} Nov 02 20:15:19.961541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:19.961465 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" event={"ID":"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8","Type":"ContainerStarted","Data":"78291b308f2169e75f6a6745cbc2a061c7372e486a425409e747b90beb154db8"} Nov 02 20:15:20.127406 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:20.127366 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz"] Nov 02 20:15:20.131309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:20.131267 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894175-sjnhz"] Nov 02 20:15:20.234068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:20.233957 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" podStartSLOduration=2.233944774 podStartE2EDuration="2.233944774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:20.232270975 +0000 UTC m=+1362.842923080" watchObservedRunningTime="2026-11-02 20:15:20.233944774 +0000 UTC m=+1362.844596878" Nov 02 20:15:20.970982 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:20.970954 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:15:21.517232 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:21.517193 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" path="/var/lib/kubelet/pods/e758972f-7e9f-4e99-9339-f41ec7c7600d/volumes" Nov 02 20:15:26.916013 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:26.915980 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:15:26.916509 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:26.916049 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:15:27.674140 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.673833 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs"] Nov 02 20:15:27.674869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.674839 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerName="cleanup" Nov 02 20:15:27.674869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.674865 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.674876 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.674884 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.674902 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.674909 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.674932 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.674939 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.674975 5399 cpu_manager.go:414] "containerMap: removing container" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.674984 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.675003 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.675010 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.675052 5399 cpu_manager.go:414] "containerMap: removing container" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerName="cleanup" Nov 02 20:15:27.675068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.675059 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerName="cleanup" Nov 02 20:15:27.676015 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.675993 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerName="cleanup" Nov 02 20:15:27.676165 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.676142 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerName="cleanup" Nov 02 20:15:27.676268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.676170 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerName="cleanup" Nov 02 20:15:27.676268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.676186 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerName="cleanup" Nov 02 20:15:27.676268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.676206 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerName="cleanup" Nov 02 20:15:27.676268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.676217 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerName="cleanup" Nov 02 20:15:27.680946 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.680918 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:15:27.735437 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.735394 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs"] Nov 02 20:15:27.771555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.771506 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2d73d56c-dockercfg-bcljf\"" Nov 02 20:15:27.771754 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.771642 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2d73d56c\"" Nov 02 20:15:27.839649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.839612 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hph\" (UniqueName: \"kubernetes.io/projected/eaea9b0b-93e7-4c63-83f4-eb8394c69272-kube-api-access-z9hph\") pod \"payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs\" (UID: \"eaea9b0b-93e7-4c63-83f4-eb8394c69272\") " pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:15:27.839857 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.839831 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaea9b0b-93e7-4c63-83f4-eb8394c69272-plugins-config-volume\") pod \"payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs\" (UID: \"eaea9b0b-93e7-4c63-83f4-eb8394c69272\") " pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:15:27.941097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.941017 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaea9b0b-93e7-4c63-83f4-eb8394c69272-plugins-config-volume\") pod \"payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs\" (UID: \"eaea9b0b-93e7-4c63-83f4-eb8394c69272\") " pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:15:27.941576 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.941180 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hph\" (UniqueName: \"kubernetes.io/projected/eaea9b0b-93e7-4c63-83f4-eb8394c69272-kube-api-access-z9hph\") pod \"payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs\" (UID: \"eaea9b0b-93e7-4c63-83f4-eb8394c69272\") " pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:15:27.942033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:27.942009 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaea9b0b-93e7-4c63-83f4-eb8394c69272-plugins-config-volume\") pod \"payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs\" (UID: \"eaea9b0b-93e7-4c63-83f4-eb8394c69272\") " pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:15:28.017403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:28.017369 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hph\" (UniqueName: \"kubernetes.io/projected/eaea9b0b-93e7-4c63-83f4-eb8394c69272-kube-api-access-z9hph\") pod \"payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs\" (UID: \"eaea9b0b-93e7-4c63-83f4-eb8394c69272\") " pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:15:28.314552 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:28.314519 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:15:28.690218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:28.690185 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs"] Nov 02 20:15:28.690540 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:15:28.690509 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaea9b0b_93e7_4c63_83f4_eb8394c69272.slice/crio-4e8ff8c8a908cb181d2ea511c6f605f849977e45855efb98a9fd3c488c1bfec5 WatchSource:0}: Error finding container 4e8ff8c8a908cb181d2ea511c6f605f849977e45855efb98a9fd3c488c1bfec5: Status 404 returned error can't find the container with id 4e8ff8c8a908cb181d2ea511c6f605f849977e45855efb98a9fd3c488c1bfec5 Nov 02 20:15:29.040937 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:29.040846 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" event={"ID":"eaea9b0b-93e7-4c63-83f4-eb8394c69272","Type":"ContainerStarted","Data":"e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001"} Nov 02 20:15:29.040937 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:29.040908 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" event={"ID":"eaea9b0b-93e7-4c63-83f4-eb8394c69272","Type":"ContainerStarted","Data":"4e8ff8c8a908cb181d2ea511c6f605f849977e45855efb98a9fd3c488c1bfec5"} Nov 02 20:15:29.040937 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:29.040930 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:15:29.148146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:29.148088 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" podStartSLOduration=2.148076359 podStartE2EDuration="2.148076359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:29.146233412 +0000 UTC m=+1371.756885484" watchObservedRunningTime="2026-11-02 20:15:29.148076359 +0000 UTC m=+1371.758728450" Nov 02 20:15:31.981975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:31.981945 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:15:33.589820 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.589781 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww"] Nov 02 20:15:33.590606 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.590587 5399 cpu_manager.go:414] "containerMap: removing container" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerName="cleanup" Nov 02 20:15:33.590682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.590608 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerName="cleanup" Nov 02 20:15:33.590682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.590624 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerName="cleanup" Nov 02 20:15:33.590682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.590630 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerName="cleanup" Nov 02 20:15:33.590780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.590737 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e758972f-7e9f-4e99-9339-f41ec7c7600d" containerName="cleanup" Nov 02 20:15:33.590780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.590748 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02079f5-208b-4062-8861-b4088cc6b3c3" containerName="cleanup" Nov 02 20:15:33.590780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.590757 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="251b41b6-5a95-4e5e-a36d-c2b2ab56cac1" containerName="cleanup" Nov 02 20:15:33.594898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.594873 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.631246 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.631218 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.631399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.631344 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.631515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.631487 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.631568 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.631542 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.631606 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.631578 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cdb\" (UniqueName: \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-kube-api-access-n4cdb\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.631713 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.631701 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.631748 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.631730 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.631782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.631754 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.631868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.631844 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.648521 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.648493 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww"] Nov 02 20:15:33.733445 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.733408 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.733639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.733459 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.733639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.733479 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.733639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.733494 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cdb\" (UniqueName: \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-kube-api-access-n4cdb\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.733639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.733596 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.733639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.733636 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.733944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.733676 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.733944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.733822 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.733944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.733892 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-credential-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.733944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.733914 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-certs\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.734171 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.734034 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.734228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.734186 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-socket\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.734332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.734318 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istiod-ca-cert\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.734445 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.734423 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-data\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.735842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.735816 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-envoy\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.736175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.736157 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-podinfo\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.786621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.786589 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-token\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.786762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.786663 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cdb\" (UniqueName: \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-kube-api-access-n4cdb\") pod \"e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:33.909835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:33.909755 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:34.331540 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:15:34.331509 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa98bd03_1a45_496f_a703_2ed86e9b62e9.slice/crio-7542ba3b57901fa76d014da68383947ef91a1ef5316243da85c29efd9c23827b WatchSource:0}: Error finding container 7542ba3b57901fa76d014da68383947ef91a1ef5316243da85c29efd9c23827b: Status 404 returned error can't find the container with id 7542ba3b57901fa76d014da68383947ef91a1ef5316243da85c29efd9c23827b Nov 02 20:15:34.332325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:34.332301 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww"] Nov 02 20:15:34.334171 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:34.334142 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:15:34.334238 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:34.334206 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:15:34.334300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:34.334243 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:15:35.095023 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:35.094983 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" event={"ID":"fa98bd03-1a45-496f-a703-2ed86e9b62e9","Type":"ContainerStarted","Data":"92716e51a8ef9c94abe72a607c23245bcbae88c074edb177c9544283fe8e08e4"} Nov 02 20:15:35.095451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:35.095047 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" event={"ID":"fa98bd03-1a45-496f-a703-2ed86e9b62e9","Type":"ContainerStarted","Data":"7542ba3b57901fa76d014da68383947ef91a1ef5316243da85c29efd9c23827b"} Nov 02 20:15:35.910652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:35.910615 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:35.912219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:35.912190 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:15:35.912360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:35.912238 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:15:36.911011 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:36.910975 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:15:36.911436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:36.911033 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:15:37.751299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:37.750627 5399 scope.go:122] "RemoveContainer" containerID="9e7f6e93008c3b27297325d52a4233258a3b01824ba1ad9c56bd00f847d45995" Nov 02 20:15:37.763043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:37.763018 5399 scope.go:122] "RemoveContainer" containerID="85e37e18e78b096676790e658f2006c73b84bda700f717bbe81e9fe9e69378f4" Nov 02 20:15:37.772442 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:37.772423 5399 scope.go:122] "RemoveContainer" containerID="b672231b17fb1d918ea5c347f7a5c142444b3cf8a8fda10d14c49e753dfcbcb4" Nov 02 20:15:37.781564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:37.781540 5399 scope.go:122] "RemoveContainer" containerID="d222259858e7ba8862e0e0e98dae7af2367dba701b1d0954f3e8be085af43a41" Nov 02 20:15:37.910532 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:37.910503 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:15:37.910670 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:37.910558 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:15:38.854298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:38.854003 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" podStartSLOduration=5.853984852 podStartE2EDuration="5.853984852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:35.299235216 +0000 UTC m=+1377.909887307" watchObservedRunningTime="2026-11-02 20:15:38.853984852 +0000 UTC m=+1381.464636944" Nov 02 20:15:38.874625 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:38.874586 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks"] Nov 02 20:15:38.881778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:38.881754 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:15:38.909566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:38.909537 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks"] Nov 02 20:15:38.910101 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:38.910078 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:15:38.910212 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:38.910130 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:15:38.921823 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:38.921794 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbsf2\" (UniqueName: \"kubernetes.io/projected/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-kube-api-access-dbsf2\") pod \"payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks\" (UID: \"37aa955b-ef9d-4016-8775-6df4bb3bcd2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:15:38.921975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:38.921944 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-plugins-config-volume\") pod \"payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks\" (UID: \"37aa955b-ef9d-4016-8775-6df4bb3bcd2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:15:38.950655 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:38.950621 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7f75722a\"" Nov 02 20:15:38.967345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:38.951648 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7f75722a-dockercfg-wsjqs\"" Nov 02 20:15:39.059481 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:39.059440 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbsf2\" (UniqueName: \"kubernetes.io/projected/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-kube-api-access-dbsf2\") pod \"payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks\" (UID: \"37aa955b-ef9d-4016-8775-6df4bb3bcd2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:15:39.065300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:39.059900 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-plugins-config-volume\") pod \"payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks\" (UID: \"37aa955b-ef9d-4016-8775-6df4bb3bcd2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:15:39.070016 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:39.069975 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-plugins-config-volume\") pod \"payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks\" (UID: \"37aa955b-ef9d-4016-8775-6df4bb3bcd2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:15:39.120717 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:39.120637 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbsf2\" (UniqueName: \"kubernetes.io/projected/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-kube-api-access-dbsf2\") pod \"payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks\" (UID: \"37aa955b-ef9d-4016-8775-6df4bb3bcd2a\") " pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:15:39.213171 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:39.213133 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:15:39.658181 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:39.658156 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks"] Nov 02 20:15:39.910840 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:39.910750 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:15:39.910840 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:39.910809 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:15:40.053532 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:40.053502 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:15:40.179372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:40.178251 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" event={"ID":"37aa955b-ef9d-4016-8775-6df4bb3bcd2a","Type":"ContainerStarted","Data":"b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8"} Nov 02 20:15:40.179372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:40.178334 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" event={"ID":"37aa955b-ef9d-4016-8775-6df4bb3bcd2a","Type":"ContainerStarted","Data":"5fb8e05791d20aaf019d70b6e52b9be33a0dbba865675bdd6ffc41b2eaad5998"} Nov 02 20:15:40.180070 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:40.180046 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:15:40.311237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:40.311181 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" podStartSLOduration=2.311167337 podStartE2EDuration="2.311167337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:40.305267312 +0000 UTC m=+1382.915919404" watchObservedRunningTime="2026-11-02 20:15:40.311167337 +0000 UTC m=+1382.921819428" Nov 02 20:15:40.910642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:40.910609 5399 patch_prober.go:29] interesting pod/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 02 20:15:40.910844 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:40.910663 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 02 20:15:41.915941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:41.915905 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:41.916375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:41.916328 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:41.917244 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:41.917221 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:15:45.267779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:45.267735 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:15:48.450301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:48.449392 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:48.487488 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:48.487435 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:48.547561 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:48.547508 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:15:48.549139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:48.549103 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 02 20:15:48.707357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:48.706791 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 02 20:15:48.806301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:48.788712 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:48.806301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:48.801521 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:15:48.899301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:48.885819 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m"] Nov 02 20:15:48.924300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:48.923398 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:48.971625 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:48.970057 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m"] Nov 02 20:15:49.010305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:48.995609 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a2b4e70c-serving-cert\"" Nov 02 20:15:49.024300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.021820 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0848b0-854e-45bd-85fe-fb952b51a9fc-openshift-service-ca\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.030306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.029961 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlzg\" (UniqueName: \"kubernetes.io/projected/9b0848b0-854e-45bd-85fe-fb952b51a9fc-kube-api-access-mtlzg\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.030306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.030179 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-metrics-certs\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.030306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.030227 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.145499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.145453 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-metrics-certs\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.145786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.145763 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.146001 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.145981 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0848b0-854e-45bd-85fe-fb952b51a9fc-openshift-service-ca\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.146317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.146292 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlzg\" (UniqueName: \"kubernetes.io/projected/9b0848b0-854e-45bd-85fe-fb952b51a9fc-kube-api-access-mtlzg\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.156301 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:49.147329 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-serving-cert: secret "maas-api-e2e-ait-a2b4e70c-serving-cert" not found Nov 02 20:15:49.156301 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:49.147401 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls podName:9b0848b0-854e-45bd-85fe-fb952b51a9fc nodeName:}" failed. No retries permitted until 2026-11-02 20:15:49.64737886 +0000 UTC m=+1392.258030948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls") pod "maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" (UID: "9b0848b0-854e-45bd-85fe-fb952b51a9fc") : secret "maas-api-e2e-ait-a2b4e70c-serving-cert" not found Nov 02 20:15:49.171737 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.148260 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0848b0-854e-45bd-85fe-fb952b51a9fc-openshift-service-ca\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.171737 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.156742 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:49.220572 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.210367 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:49.263505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.263399 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:49.326057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.297996 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlzg\" (UniqueName: \"kubernetes.io/projected/9b0848b0-854e-45bd-85fe-fb952b51a9fc-kube-api-access-mtlzg\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.349180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.349141 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-metrics-certs\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.468451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.468403 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:15:49.469818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.469791 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:15:49.492493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.492432 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:49.494545 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.494520 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:15:49.718326 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.717719 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:49.718326 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:49.718006 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-serving-cert: secret "maas-api-e2e-ait-a2b4e70c-serving-cert" not found Nov 02 20:15:49.718326 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:49.718084 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls podName:9b0848b0-854e-45bd-85fe-fb952b51a9fc nodeName:}" failed. No retries permitted until 2026-11-02 20:15:50.718063485 +0000 UTC m=+1393.328715568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls") pod "maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" (UID: "9b0848b0-854e-45bd-85fe-fb952b51a9fc") : secret "maas-api-e2e-ait-a2b4e70c-serving-cert" not found Nov 02 20:15:49.726497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.726440 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:49.729993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.729025 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:15:49.823301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.810088 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:49.823301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.810803 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:15:49.823301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.812305 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:15:49.823301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:49.812468 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:15:50.138338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.122946 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:15:50.213428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.213378 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:15:50.214960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.214936 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:15:50.443438 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.443229 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:15:50.452502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.451376 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:15:50.452502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.451449 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:15:50.483332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.478053 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:15:50.500306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.484927 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:15:50.753967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.753879 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:50.756577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.756547 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls\") pod \"maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:50.822178 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:50.822148 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:51.190815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:51.190785 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:15:51.266676 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:15:51.266521 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0848b0_854e_45bd_85fe_fb952b51a9fc.slice/crio-15c5455678763e0edf5f24102be9625ae1a0728d90ee918cb45ccce7b64236ca WatchSource:0}: Error finding container 15c5455678763e0edf5f24102be9625ae1a0728d90ee918cb45ccce7b64236ca: Status 404 returned error can't find the container with id 15c5455678763e0edf5f24102be9625ae1a0728d90ee918cb45ccce7b64236ca Nov 02 20:15:51.272636 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:51.269686 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m"] Nov 02 20:15:51.732999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:51.732963 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" event={"ID":"9b0848b0-854e-45bd-85fe-fb952b51a9fc","Type":"ContainerStarted","Data":"15c5455678763e0edf5f24102be9625ae1a0728d90ee918cb45ccce7b64236ca"} Nov 02 20:15:52.742522 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:52.742484 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" event={"ID":"9b0848b0-854e-45bd-85fe-fb952b51a9fc","Type":"ContainerStarted","Data":"03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe"} Nov 02 20:15:52.742936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:52.742599 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:15:52.849225 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:52.849173 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" podStartSLOduration=3.850888403 podStartE2EDuration="4.849159416s" totalImagesPullingTime="998.271013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:52.848517048 +0000 UTC m=+1395.459169140" watchObservedRunningTime="2026-11-02 20:15:52.849159416 +0000 UTC m=+1395.459811507" Nov 02 20:15:53.873253 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:53.873217 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz"] Nov 02 20:15:53.877652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:53.877632 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:53.931873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:53.931673 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz"] Nov 02 20:15:53.972478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:53.972440 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-2ac9784d-serving-cert\"" Nov 02 20:15:54.003355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.003325 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-maas-api-tls\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.003532 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.003490 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-metrics-certs\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.003587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.003548 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmjv\" (UniqueName: \"kubernetes.io/projected/a550c407-5174-4765-a7ec-6b1aeae2080b-kube-api-access-rnmjv\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.003762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.003736 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a550c407-5174-4765-a7ec-6b1aeae2080b-openshift-service-ca\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.104953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.104916 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-maas-api-tls\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.105130 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.105050 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-metrics-certs\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.105130 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:54.105076 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-serving-cert: secret "maas-api-e2e-migrate-2ac9784d-serving-cert" not found Nov 02 20:15:54.105130 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.105104 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmjv\" (UniqueName: \"kubernetes.io/projected/a550c407-5174-4765-a7ec-6b1aeae2080b-kube-api-access-rnmjv\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.105251 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:15:54.105145 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-maas-api-tls podName:a550c407-5174-4765-a7ec-6b1aeae2080b nodeName:}" failed. No retries permitted until 2026-11-02 20:15:54.605122529 +0000 UTC m=+1397.215774620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-maas-api-tls") pod "maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" (UID: "a550c407-5174-4765-a7ec-6b1aeae2080b") : secret "maas-api-e2e-migrate-2ac9784d-serving-cert" not found Nov 02 20:15:54.105332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.105267 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a550c407-5174-4765-a7ec-6b1aeae2080b-openshift-service-ca\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.106058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.106030 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a550c407-5174-4765-a7ec-6b1aeae2080b-openshift-service-ca\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.108248 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.108200 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-metrics-certs\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.161774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.161699 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmjv\" (UniqueName: \"kubernetes.io/projected/a550c407-5174-4765-a7ec-6b1aeae2080b-kube-api-access-rnmjv\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.613607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.613539 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-maas-api-tls\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.616036 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.616009 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-maas-api-tls\") pod \"maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:54.799331 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:54.799298 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:55.172813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:55.172783 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz"] Nov 02 20:15:55.176100 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:15:55.176062 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda550c407_5174_4765_a7ec_6b1aeae2080b.slice/crio-d6757bc5966fedd29484216d0efa0c913f9cfc532395492d1e590798aa2170ce WatchSource:0}: Error finding container d6757bc5966fedd29484216d0efa0c913f9cfc532395492d1e590798aa2170ce: Status 404 returned error can't find the container with id d6757bc5966fedd29484216d0efa0c913f9cfc532395492d1e590798aa2170ce Nov 02 20:15:55.768124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:55.768089 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" event={"ID":"a550c407-5174-4765-a7ec-6b1aeae2080b","Type":"ContainerStarted","Data":"d6757bc5966fedd29484216d0efa0c913f9cfc532395492d1e590798aa2170ce"} Nov 02 20:15:56.785973 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:56.785935 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" event={"ID":"a550c407-5174-4765-a7ec-6b1aeae2080b","Type":"ContainerStarted","Data":"1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f"} Nov 02 20:15:56.786495 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:56.786075 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:15:56.896709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:56.896636 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" podStartSLOduration=2.999770159 podStartE2EDuration="3.896617725s" totalImagesPullingTime="896.847566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:56.892548697 +0000 UTC m=+1399.503200788" watchObservedRunningTime="2026-11-02 20:15:56.896617725 +0000 UTC m=+1399.507269819" Nov 02 20:15:58.756736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:15:58.756705 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:16:02.799529 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:02.799498 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:16:11.862842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:11.862807 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp"] Nov 02 20:16:11.868702 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:11.868681 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:11.919868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:11.919833 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp"] Nov 02 20:16:11.960612 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:11.960575 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2d73d56c-serving-cert\"" Nov 02 20:16:11.986950 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:11.986911 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9vbk\" (UniqueName: \"kubernetes.io/projected/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-kube-api-access-p9vbk\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:11.987146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:11.986980 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:11.987146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:11.987123 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-openshift-service-ca\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:11.987243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:11.987229 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-metrics-certs\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:12.089117 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:12.089074 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9vbk\" (UniqueName: \"kubernetes.io/projected/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-kube-api-access-p9vbk\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:12.089315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:12.089169 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:12.089315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:12.089303 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-openshift-service-ca\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:12.089456 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:16:12.089349 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-serving-cert: secret "maas-api-e2e-derive-2d73d56c-serving-cert" not found Nov 02 20:16:12.089456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:12.089392 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-metrics-certs\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:12.089456 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:16:12.089446 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls podName:92e5d4f1-2e2b-44f2-8732-cf8fec071edb nodeName:}" failed. No retries permitted until 2026-11-02 20:16:12.589424363 +0000 UTC m=+1415.200076449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls") pod "maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" (UID: "92e5d4f1-2e2b-44f2-8732-cf8fec071edb") : secret "maas-api-e2e-derive-2d73d56c-serving-cert" not found Nov 02 20:16:12.090265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:12.090230 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-openshift-service-ca\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:12.092040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:12.092014 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-metrics-certs\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:12.144990 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:12.144908 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9vbk\" (UniqueName: \"kubernetes.io/projected/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-kube-api-access-p9vbk\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:12.596824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:12.596782 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:12.597007 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:16:12.596929 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-serving-cert: secret "maas-api-e2e-derive-2d73d56c-serving-cert" not found Nov 02 20:16:12.597007 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:16:12.596993 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls podName:92e5d4f1-2e2b-44f2-8732-cf8fec071edb nodeName:}" failed. No retries permitted until 2026-11-02 20:16:13.596975899 +0000 UTC m=+1416.207627969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls") pod "maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" (UID: "92e5d4f1-2e2b-44f2-8732-cf8fec071edb") : secret "maas-api-e2e-derive-2d73d56c-serving-cert" not found Nov 02 20:16:13.616081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:13.616045 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:13.618650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:13.618623 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls\") pod \"maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:13.691797 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:13.691757 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:14.075390 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:14.075351 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp"] Nov 02 20:16:14.075658 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:16:14.075631 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e5d4f1_2e2b_44f2_8732_cf8fec071edb.slice/crio-c446546e4c3be64e65d547fb71e6d8acf684cf80c059a006e80662a8895c7f0f WatchSource:0}: Error finding container c446546e4c3be64e65d547fb71e6d8acf684cf80c059a006e80662a8895c7f0f: Status 404 returned error can't find the container with id c446546e4c3be64e65d547fb71e6d8acf684cf80c059a006e80662a8895c7f0f Nov 02 20:16:14.925074 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:14.925037 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" event={"ID":"92e5d4f1-2e2b-44f2-8732-cf8fec071edb","Type":"ContainerStarted","Data":"c446546e4c3be64e65d547fb71e6d8acf684cf80c059a006e80662a8895c7f0f"} Nov 02 20:16:15.934054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:15.934015 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" event={"ID":"92e5d4f1-2e2b-44f2-8732-cf8fec071edb","Type":"ContainerStarted","Data":"7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2"} Nov 02 20:16:15.934558 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:15.934114 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:16.060140 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:16.060074 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" podStartSLOduration=4.11061767 podStartE2EDuration="5.060056791s" totalImagesPullingTime="949.439121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:16.05586559 +0000 UTC m=+1418.666517679" watchObservedRunningTime="2026-11-02 20:16:16.060056791 +0000 UTC m=+1418.670708883" Nov 02 20:16:18.283949 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:16:18.283906 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:16:21.992445 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:21.992414 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:16:26.059419 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:26.059381 5399 generic.go:356] "Generic (PLEG): container finished" podID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerID="56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92" exitCode=0 Nov 02 20:16:26.060570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:26.059455 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" event={"ID":"d83953ff-1dd7-4b12-bc48-90ef3658777d","Type":"ContainerDied","Data":"56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92"} Nov 02 20:16:26.060570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:26.059895 5399 scope.go:122] "RemoveContainer" containerID="56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92" Nov 02 20:16:26.061606 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:26.061558 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:16:26.061769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:26.061658 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:16:26.061769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:26.061706 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:16:27.069644 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:27.069603 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" event={"ID":"d83953ff-1dd7-4b12-bc48-90ef3658777d","Type":"ContainerStarted","Data":"abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250"} Nov 02 20:16:27.703547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:27.703518 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:16:27.703725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:27.703635 5399 patch_prober.go:29] interesting pod/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 20:16:27.703725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:27.703691 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 20:16:28.704353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:28.704320 5399 patch_prober.go:29] interesting pod/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 20:16:28.704759 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:28.704372 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 20:16:29.704506 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:29.704472 5399 patch_prober.go:29] interesting pod/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 20:16:29.704892 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:29.704539 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 20:16:30.222215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.222181 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42"] Nov 02 20:16:30.243422 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.242559 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.280206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.280172 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42"] Nov 02 20:16:30.318630 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.318594 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3507784-bbfa-439c-bc46-e2e62076cc3a-openshift-service-ca\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.318890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.318868 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d48jh\" (UniqueName: \"kubernetes.io/projected/b3507784-bbfa-439c-bc46-e2e62076cc3a-kube-api-access-d48jh\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.319105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.319086 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-maas-api-tls\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.319288 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.319257 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-metrics-certs\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.320765 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.320738 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7f75722a-serving-cert\"" Nov 02 20:16:30.420927 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.420894 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3507784-bbfa-439c-bc46-e2e62076cc3a-openshift-service-ca\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.421094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.420939 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d48jh\" (UniqueName: \"kubernetes.io/projected/b3507784-bbfa-439c-bc46-e2e62076cc3a-kube-api-access-d48jh\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.421094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.421001 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-maas-api-tls\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.421094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.421035 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-metrics-certs\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.421212 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:16:30.421170 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-serving-cert: secret "maas-api-e2e-mt-7f75722a-serving-cert" not found Nov 02 20:16:30.421253 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:16:30.421232 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-maas-api-tls podName:b3507784-bbfa-439c-bc46-e2e62076cc3a nodeName:}" failed. No retries permitted until 2026-11-02 20:16:30.921216647 +0000 UTC m=+1433.531868733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-maas-api-tls") pod "maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" (UID: "b3507784-bbfa-439c-bc46-e2e62076cc3a") : secret "maas-api-e2e-mt-7f75722a-serving-cert" not found Nov 02 20:16:30.421659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.421640 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3507784-bbfa-439c-bc46-e2e62076cc3a-openshift-service-ca\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.423382 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.423364 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-metrics-certs\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.487050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.486953 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d48jh\" (UniqueName: \"kubernetes.io/projected/b3507784-bbfa-439c-bc46-e2e62076cc3a-kube-api-access-d48jh\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.703786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.703750 5399 patch_prober.go:29] interesting pod/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 20:16:30.703993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.703809 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 20:16:30.951578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.951535 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-maas-api-tls\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:30.955030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:30.955000 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-maas-api-tls\") pod \"maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:31.166725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:31.166698 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:31.557253 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:31.557218 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42"] Nov 02 20:16:31.558097 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:16:31.558073 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3507784_bbfa_439c_bc46_e2e62076cc3a.slice/crio-73d8b8a6f747a1022de21f3347c78d256b50bdd89d09929e27b8561cc21f7b0a WatchSource:0}: Error finding container 73d8b8a6f747a1022de21f3347c78d256b50bdd89d09929e27b8561cc21f7b0a: Status 404 returned error can't find the container with id 73d8b8a6f747a1022de21f3347c78d256b50bdd89d09929e27b8561cc21f7b0a Nov 02 20:16:31.704081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:31.704042 5399 patch_prober.go:29] interesting pod/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 02 20:16:31.704262 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:31.704107 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 02 20:16:32.109352 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:32.109316 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" event={"ID":"b3507784-bbfa-439c-bc46-e2e62076cc3a","Type":"ContainerStarted","Data":"73d8b8a6f747a1022de21f3347c78d256b50bdd89d09929e27b8561cc21f7b0a"} Nov 02 20:16:32.708965 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:32.708877 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:16:32.709408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:32.709390 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:16:32.710188 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:32.710165 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:16:33.138513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:33.138347 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" event={"ID":"b3507784-bbfa-439c-bc46-e2e62076cc3a","Type":"ContainerStarted","Data":"96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04"} Nov 02 20:16:33.138513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:33.138438 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:33.248422 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:33.248357 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" podStartSLOduration=2.413377701 podStartE2EDuration="3.248336492s" totalImagesPullingTime="834.958791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:33.243600246 +0000 UTC m=+1435.854252336" watchObservedRunningTime="2026-11-02 20:16:33.248336492 +0000 UTC m=+1435.858988600" Nov 02 20:16:39.152596 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:39.152558 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:16:48.244295 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:16:48.244181 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:16:52.814141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:16:52.814101 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:17:05.624460 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:05.624422 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29"] Nov 02 20:17:05.740669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:05.740633 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz"] Nov 02 20:17:08.563858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:08.563817 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8"] Nov 02 20:17:10.625654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:10.625592 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" podUID="c894a85f-10e2-44d9-993b-c50b43ad9734" containerName="payload-processing" containerID="cri-o://77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b" gracePeriod=25 Nov 02 20:17:10.742042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:10.741984 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" podUID="7c044378-c69b-45cb-893c-0d98abf6cb85" containerName="payload-pre-processing" containerID="cri-o://4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44" gracePeriod=25 Nov 02 20:17:10.950204 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:10.949852 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:17:10.989856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:10.988765 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c894a85f-10e2-44d9-993b-c50b43ad9734-kube-api-access-njt5h\" (UniqueName: \"kubernetes.io/projected/c894a85f-10e2-44d9-993b-c50b43ad9734-kube-api-access-njt5h\") pod \"c894a85f-10e2-44d9-993b-c50b43ad9734\" (UID: \"c894a85f-10e2-44d9-993b-c50b43ad9734\") " Nov 02 20:17:10.989856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:10.988967 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c894a85f-10e2-44d9-993b-c50b43ad9734-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c894a85f-10e2-44d9-993b-c50b43ad9734-plugins-config-volume\") pod \"c894a85f-10e2-44d9-993b-c50b43ad9734\" (UID: \"c894a85f-10e2-44d9-993b-c50b43ad9734\") " Nov 02 20:17:10.990767 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:10.990688 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c894a85f-10e2-44d9-993b-c50b43ad9734-plugins-config-volume" pod "c894a85f-10e2-44d9-993b-c50b43ad9734" (UID: "c894a85f-10e2-44d9-993b-c50b43ad9734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:11.004342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.001557 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c894a85f-10e2-44d9-993b-c50b43ad9734-kube-api-access-njt5h" pod "c894a85f-10e2-44d9-993b-c50b43ad9734" (UID: "c894a85f-10e2-44d9-993b-c50b43ad9734"). InnerVolumeSpecName "kube-api-access-njt5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:11.059814 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.059782 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:17:11.091075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.091033 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c044378-c69b-45cb-893c-0d98abf6cb85-kube-api-access-h4g87\" (UniqueName: \"kubernetes.io/projected/7c044378-c69b-45cb-893c-0d98abf6cb85-kube-api-access-h4g87\") pod \"7c044378-c69b-45cb-893c-0d98abf6cb85\" (UID: \"7c044378-c69b-45cb-893c-0d98abf6cb85\") " Nov 02 20:17:11.091264 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.091163 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c044378-c69b-45cb-893c-0d98abf6cb85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c044378-c69b-45cb-893c-0d98abf6cb85-plugins-config-volume\") pod \"7c044378-c69b-45cb-893c-0d98abf6cb85\" (UID: \"7c044378-c69b-45cb-893c-0d98abf6cb85\") " Nov 02 20:17:11.091997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.091967 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c894a85f-10e2-44d9-993b-c50b43ad9734-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:11.091997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.091999 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njt5h\" (UniqueName: \"kubernetes.io/projected/c894a85f-10e2-44d9-993b-c50b43ad9734-kube-api-access-njt5h\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:11.092226 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.091990 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c044378-c69b-45cb-893c-0d98abf6cb85-plugins-config-volume" pod "7c044378-c69b-45cb-893c-0d98abf6cb85" (UID: "7c044378-c69b-45cb-893c-0d98abf6cb85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:11.094407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.094379 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c044378-c69b-45cb-893c-0d98abf6cb85-kube-api-access-h4g87" pod "7c044378-c69b-45cb-893c-0d98abf6cb85" (UID: "7c044378-c69b-45cb-893c-0d98abf6cb85"). InnerVolumeSpecName "kube-api-access-h4g87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:11.193839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.193746 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4g87\" (UniqueName: \"kubernetes.io/projected/7c044378-c69b-45cb-893c-0d98abf6cb85-kube-api-access-h4g87\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:11.194061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.194041 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c044378-c69b-45cb-893c-0d98abf6cb85-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:11.509049 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.508957 5399 generic.go:356] "Generic (PLEG): container finished" podID="7c044378-c69b-45cb-893c-0d98abf6cb85" containerID="4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44" exitCode=0 Nov 02 20:17:11.509401 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.509382 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" Nov 02 20:17:11.519673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.519642 5399 generic.go:356] "Generic (PLEG): container finished" podID="c894a85f-10e2-44d9-993b-c50b43ad9734" containerID="77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b" exitCode=0 Nov 02 20:17:11.520166 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.520146 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" Nov 02 20:17:11.530689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.530658 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" event={"ID":"7c044378-c69b-45cb-893c-0d98abf6cb85","Type":"ContainerDied","Data":"4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44"} Nov 02 20:17:11.530808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.530723 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz" event={"ID":"7c044378-c69b-45cb-893c-0d98abf6cb85","Type":"ContainerDied","Data":"6d3ceddff156c96daad1e8c2aa31d0fd7238ac64708ff2291770f23fbafe65fb"} Nov 02 20:17:11.530808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.530748 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" event={"ID":"c894a85f-10e2-44d9-993b-c50b43ad9734","Type":"ContainerDied","Data":"77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b"} Nov 02 20:17:11.530808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.530774 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29" event={"ID":"c894a85f-10e2-44d9-993b-c50b43ad9734","Type":"ContainerDied","Data":"d56c19a31b63e5bde8d8cc03926d42d1b9c91cc9af68b5ce4f3aad344f444cfd"} Nov 02 20:17:11.530808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.530803 5399 scope.go:122] "RemoveContainer" containerID="4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44" Nov 02 20:17:11.550015 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.549991 5399 scope.go:122] "RemoveContainer" containerID="4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44" Nov 02 20:17:11.550422 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:11.550398 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44\": container with ID starting with 4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44 not found: ID does not exist" containerID="4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44" Nov 02 20:17:11.550593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.550558 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44"} err="failed to get container status \"4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44\": rpc error: code = NotFound desc = could not find container \"4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44\": container with ID starting with 4355c1ebefb7c7f55627dbfa8144798ff40d2e42635123aa236127dfcc822d44 not found: ID does not exist" Nov 02 20:17:11.550694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.550596 5399 scope.go:122] "RemoveContainer" containerID="77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b" Nov 02 20:17:11.563196 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.563154 5399 scope.go:122] "RemoveContainer" containerID="77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b" Nov 02 20:17:11.563558 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:11.563537 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b\": container with ID starting with 77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b not found: ID does not exist" containerID="77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b" Nov 02 20:17:11.563642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.563568 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b"} err="failed to get container status \"77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b\": rpc error: code = NotFound desc = could not find container \"77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b\": container with ID starting with 77d0cb3e8e3bdf640cfc9f0409a452caf9b9beb7d8f3d066293f983cb1c0753b not found: ID does not exist" Nov 02 20:17:11.690651 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.690611 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz"] Nov 02 20:17:11.695673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.695637 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2ac9784d-74469b6b75-ckhsz"] Nov 02 20:17:11.779504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.779419 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29"] Nov 02 20:17:11.783825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:11.783795 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2ac9784d-c4d669545-m5j29"] Nov 02 20:17:13.509423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:13.509391 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c044378-c69b-45cb-893c-0d98abf6cb85" path="/var/lib/kubelet/pods/7c044378-c69b-45cb-893c-0d98abf6cb85/volumes" Nov 02 20:17:13.510505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:13.510478 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c894a85f-10e2-44d9-993b-c50b43ad9734" path="/var/lib/kubelet/pods/c894a85f-10e2-44d9-993b-c50b43ad9734/volumes" Nov 02 20:17:13.564497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:13.564452 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" podUID="dccd1cf9-eaa9-4f2c-b018-8346316cdfb8" containerName="payload-processing" containerID="cri-o://d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001" gracePeriod=25 Nov 02 20:17:13.856877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:13.856849 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:17:13.946796 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:13.946761 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-kube-api-access-vnbk2\" (UniqueName: \"kubernetes.io/projected/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-kube-api-access-vnbk2\") pod \"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8\" (UID: \"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8\") " Nov 02 20:17:13.946962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:13.946888 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-plugins-config-volume\") pod \"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8\" (UID: \"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8\") " Nov 02 20:17:13.947263 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:13.947232 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-plugins-config-volume" pod "dccd1cf9-eaa9-4f2c-b018-8346316cdfb8" (UID: "dccd1cf9-eaa9-4f2c-b018-8346316cdfb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:13.947474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:13.947459 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:13.948972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:13.948948 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-kube-api-access-vnbk2" pod "dccd1cf9-eaa9-4f2c-b018-8346316cdfb8" (UID: "dccd1cf9-eaa9-4f2c-b018-8346316cdfb8"). InnerVolumeSpecName "kube-api-access-vnbk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:14.049235 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:14.049191 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnbk2\" (UniqueName: \"kubernetes.io/projected/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8-kube-api-access-vnbk2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:14.546298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:14.546248 5399 generic.go:356] "Generic (PLEG): container finished" podID="dccd1cf9-eaa9-4f2c-b018-8346316cdfb8" containerID="d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001" exitCode=0 Nov 02 20:17:14.546771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:14.546320 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" Nov 02 20:17:14.546771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:14.546318 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" event={"ID":"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8","Type":"ContainerDied","Data":"d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001"} Nov 02 20:17:14.546771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:14.546433 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8" event={"ID":"dccd1cf9-eaa9-4f2c-b018-8346316cdfb8","Type":"ContainerDied","Data":"78291b308f2169e75f6a6745cbc2a061c7372e486a425409e747b90beb154db8"} Nov 02 20:17:14.546771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:14.546455 5399 scope.go:122] "RemoveContainer" containerID="d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001" Nov 02 20:17:14.561780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:14.561751 5399 scope.go:122] "RemoveContainer" containerID="d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001" Nov 02 20:17:14.562237 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:14.562217 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001\": container with ID starting with d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001 not found: ID does not exist" containerID="d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001" Nov 02 20:17:14.562351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:14.562247 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001"} err="failed to get container status \"d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001\": rpc error: code = NotFound desc = could not find container \"d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001\": container with ID starting with d4926e4c6b934ec4ccac270e760bba082422807a75ec66d5334b8bfa31384001 not found: ID does not exist" Nov 02 20:17:14.703594 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:14.703558 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8"] Nov 02 20:17:14.749338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:14.749306 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2b4e70c-589f7dcbc9-qrgj8"] Nov 02 20:17:15.511154 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:15.511114 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccd1cf9-eaa9-4f2c-b018-8346316cdfb8" path="/var/lib/kubelet/pods/dccd1cf9-eaa9-4f2c-b018-8346316cdfb8/volumes" Nov 02 20:17:18.238244 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:18.238201 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:17:27.889485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:27.889446 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs"] Nov 02 20:17:32.890722 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:32.890674 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" podUID="eaea9b0b-93e7-4c63-83f4-eb8394c69272" containerName="payload-processing" containerID="cri-o://e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001" gracePeriod=25 Nov 02 20:17:33.200568 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.200537 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:17:33.259579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.259548 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaea9b0b-93e7-4c63-83f4-eb8394c69272-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaea9b0b-93e7-4c63-83f4-eb8394c69272-plugins-config-volume\") pod \"eaea9b0b-93e7-4c63-83f4-eb8394c69272\" (UID: \"eaea9b0b-93e7-4c63-83f4-eb8394c69272\") " Nov 02 20:17:33.259785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.259773 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaea9b0b-93e7-4c63-83f4-eb8394c69272-kube-api-access-z9hph\" (UniqueName: \"kubernetes.io/projected/eaea9b0b-93e7-4c63-83f4-eb8394c69272-kube-api-access-z9hph\") pod \"eaea9b0b-93e7-4c63-83f4-eb8394c69272\" (UID: \"eaea9b0b-93e7-4c63-83f4-eb8394c69272\") " Nov 02 20:17:33.260020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.259993 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaea9b0b-93e7-4c63-83f4-eb8394c69272-plugins-config-volume" pod "eaea9b0b-93e7-4c63-83f4-eb8394c69272" (UID: "eaea9b0b-93e7-4c63-83f4-eb8394c69272"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:33.260477 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.260456 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaea9b0b-93e7-4c63-83f4-eb8394c69272-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:33.262008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.261982 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaea9b0b-93e7-4c63-83f4-eb8394c69272-kube-api-access-z9hph" pod "eaea9b0b-93e7-4c63-83f4-eb8394c69272" (UID: "eaea9b0b-93e7-4c63-83f4-eb8394c69272"). InnerVolumeSpecName "kube-api-access-z9hph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:33.361666 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.361626 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9hph\" (UniqueName: \"kubernetes.io/projected/eaea9b0b-93e7-4c63-83f4-eb8394c69272-kube-api-access-z9hph\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:33.699876 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.699833 5399 generic.go:356] "Generic (PLEG): container finished" podID="eaea9b0b-93e7-4c63-83f4-eb8394c69272" containerID="e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001" exitCode=0 Nov 02 20:17:33.700043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.699882 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" Nov 02 20:17:33.700043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.699908 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" event={"ID":"eaea9b0b-93e7-4c63-83f4-eb8394c69272","Type":"ContainerDied","Data":"e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001"} Nov 02 20:17:33.700043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.699963 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs" event={"ID":"eaea9b0b-93e7-4c63-83f4-eb8394c69272","Type":"ContainerDied","Data":"4e8ff8c8a908cb181d2ea511c6f605f849977e45855efb98a9fd3c488c1bfec5"} Nov 02 20:17:33.700043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.699987 5399 scope.go:122] "RemoveContainer" containerID="e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001" Nov 02 20:17:33.711052 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.711034 5399 scope.go:122] "RemoveContainer" containerID="e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001" Nov 02 20:17:33.711356 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:33.711334 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001\": container with ID starting with e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001 not found: ID does not exist" containerID="e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001" Nov 02 20:17:33.711404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.711366 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001"} err="failed to get container status \"e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001\": rpc error: code = NotFound desc = could not find container \"e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001\": container with ID starting with e619279d0d6815fcf8610f7195af5900b2480191389afaf732ff04c395429001 not found: ID does not exist" Nov 02 20:17:33.856621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.856585 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs"] Nov 02 20:17:33.902820 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:33.902781 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2d73d56c-55f64cccb7-vrdzs"] Nov 02 20:17:35.509711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:35.509679 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaea9b0b-93e7-4c63-83f4-eb8394c69272" path="/var/lib/kubelet/pods/eaea9b0b-93e7-4c63-83f4-eb8394c69272/volumes" Nov 02 20:17:39.918915 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:39.918879 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m"] Nov 02 20:17:39.919353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:39.919095 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" podUID="9b0848b0-854e-45bd-85fe-fb952b51a9fc" containerName="maas-api" containerID="cri-o://03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe" gracePeriod=30 Nov 02 20:17:40.212177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.212147 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:17:40.377353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.377309 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls\") pod \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " Nov 02 20:17:40.377519 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.377388 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-metrics-certs\") pod \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " Nov 02 20:17:40.377562 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.377518 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0848b0-854e-45bd-85fe-fb952b51a9fc-kube-api-access-mtlzg\" (UniqueName: \"kubernetes.io/projected/9b0848b0-854e-45bd-85fe-fb952b51a9fc-kube-api-access-mtlzg\") pod \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " Nov 02 20:17:40.377730 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.377712 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b0848b0-854e-45bd-85fe-fb952b51a9fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0848b0-854e-45bd-85fe-fb952b51a9fc-openshift-service-ca\") pod \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\" (UID: \"9b0848b0-854e-45bd-85fe-fb952b51a9fc\") " Nov 02 20:17:40.378094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.378063 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0848b0-854e-45bd-85fe-fb952b51a9fc-openshift-service-ca" pod "9b0848b0-854e-45bd-85fe-fb952b51a9fc" (UID: "9b0848b0-854e-45bd-85fe-fb952b51a9fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:40.378425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.378404 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0848b0-854e-45bd-85fe-fb952b51a9fc-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:40.379538 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.379508 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls" pod "9b0848b0-854e-45bd-85fe-fb952b51a9fc" (UID: "9b0848b0-854e-45bd-85fe-fb952b51a9fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:40.379634 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.379550 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0848b0-854e-45bd-85fe-fb952b51a9fc-kube-api-access-mtlzg" pod "9b0848b0-854e-45bd-85fe-fb952b51a9fc" (UID: "9b0848b0-854e-45bd-85fe-fb952b51a9fc"). InnerVolumeSpecName "kube-api-access-mtlzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:40.379634 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.379567 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-metrics-certs" pod "9b0848b0-854e-45bd-85fe-fb952b51a9fc" (UID: "9b0848b0-854e-45bd-85fe-fb952b51a9fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:40.479995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.479908 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:40.479995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.479945 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0848b0-854e-45bd-85fe-fb952b51a9fc-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:40.479995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.479961 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtlzg\" (UniqueName: \"kubernetes.io/projected/9b0848b0-854e-45bd-85fe-fb952b51a9fc-kube-api-access-mtlzg\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:40.857235 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.857198 5399 generic.go:356] "Generic (PLEG): container finished" podID="9b0848b0-854e-45bd-85fe-fb952b51a9fc" containerID="03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe" exitCode=0 Nov 02 20:17:40.857458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.857250 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" Nov 02 20:17:40.857458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.857329 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" event={"ID":"9b0848b0-854e-45bd-85fe-fb952b51a9fc","Type":"ContainerDied","Data":"03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe"} Nov 02 20:17:40.857458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.857396 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m" event={"ID":"9b0848b0-854e-45bd-85fe-fb952b51a9fc","Type":"ContainerDied","Data":"15c5455678763e0edf5f24102be9625ae1a0728d90ee918cb45ccce7b64236ca"} Nov 02 20:17:40.857458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.857423 5399 scope.go:122] "RemoveContainer" containerID="03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe" Nov 02 20:17:40.871528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.871499 5399 scope.go:122] "RemoveContainer" containerID="03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe" Nov 02 20:17:40.871922 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:40.871895 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe\": container with ID starting with 03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe not found: ID does not exist" containerID="03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe" Nov 02 20:17:40.872027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.871931 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe"} err="failed to get container status \"03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe\": rpc error: code = NotFound desc = could not find container \"03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe\": container with ID starting with 03075843471d3929458b5ebe2ee8c83fa65e4ba9f50dd37a415403265ae1a1fe not found: ID does not exist" Nov 02 20:17:41.001336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.999008 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz"] Nov 02 20:17:41.001336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:40.999303 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" podUID="a550c407-5174-4765-a7ec-6b1aeae2080b" containerName="maas-api" containerID="cri-o://1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f" gracePeriod=30 Nov 02 20:17:41.014076 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.014045 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m"] Nov 02 20:17:41.050938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.050561 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2b4e70c-7b877d9fb9-5969m"] Nov 02 20:17:41.339840 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.339809 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:17:41.496843 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.496753 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-maas-api-tls\") pod \"a550c407-5174-4765-a7ec-6b1aeae2080b\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " Nov 02 20:17:41.497099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.497080 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a550c407-5174-4765-a7ec-6b1aeae2080b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a550c407-5174-4765-a7ec-6b1aeae2080b-openshift-service-ca\") pod \"a550c407-5174-4765-a7ec-6b1aeae2080b\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " Nov 02 20:17:41.497310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.497293 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a550c407-5174-4765-a7ec-6b1aeae2080b-kube-api-access-rnmjv\" (UniqueName: \"kubernetes.io/projected/a550c407-5174-4765-a7ec-6b1aeae2080b-kube-api-access-rnmjv\") pod \"a550c407-5174-4765-a7ec-6b1aeae2080b\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " Nov 02 20:17:41.497426 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.497413 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-metrics-certs\") pod \"a550c407-5174-4765-a7ec-6b1aeae2080b\" (UID: \"a550c407-5174-4765-a7ec-6b1aeae2080b\") " Nov 02 20:17:41.502941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.502905 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a550c407-5174-4765-a7ec-6b1aeae2080b-openshift-service-ca" pod "a550c407-5174-4765-a7ec-6b1aeae2080b" (UID: "a550c407-5174-4765-a7ec-6b1aeae2080b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:41.507369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.506481 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-metrics-certs" pod "a550c407-5174-4765-a7ec-6b1aeae2080b" (UID: "a550c407-5174-4765-a7ec-6b1aeae2080b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:41.510708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.510672 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a550c407-5174-4765-a7ec-6b1aeae2080b-kube-api-access-rnmjv" pod "a550c407-5174-4765-a7ec-6b1aeae2080b" (UID: "a550c407-5174-4765-a7ec-6b1aeae2080b"). InnerVolumeSpecName "kube-api-access-rnmjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:41.512955 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.512905 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-maas-api-tls" pod "a550c407-5174-4765-a7ec-6b1aeae2080b" (UID: "a550c407-5174-4765-a7ec-6b1aeae2080b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:41.519593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.519083 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0848b0-854e-45bd-85fe-fb952b51a9fc" path="/var/lib/kubelet/pods/9b0848b0-854e-45bd-85fe-fb952b51a9fc/volumes" Nov 02 20:17:41.600193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.600157 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:41.600432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.600208 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a550c407-5174-4765-a7ec-6b1aeae2080b-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:41.600432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.600235 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmjv\" (UniqueName: \"kubernetes.io/projected/a550c407-5174-4765-a7ec-6b1aeae2080b-kube-api-access-rnmjv\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:41.600432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.600252 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a550c407-5174-4765-a7ec-6b1aeae2080b-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:41.868843 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.868803 5399 generic.go:356] "Generic (PLEG): container finished" podID="a550c407-5174-4765-a7ec-6b1aeae2080b" containerID="1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f" exitCode=0 Nov 02 20:17:41.869252 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.869117 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" event={"ID":"a550c407-5174-4765-a7ec-6b1aeae2080b","Type":"ContainerDied","Data":"1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f"} Nov 02 20:17:41.869252 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.869173 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" Nov 02 20:17:41.869252 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.869189 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz" event={"ID":"a550c407-5174-4765-a7ec-6b1aeae2080b","Type":"ContainerDied","Data":"d6757bc5966fedd29484216d0efa0c913f9cfc532395492d1e590798aa2170ce"} Nov 02 20:17:41.869252 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.869221 5399 scope.go:122] "RemoveContainer" containerID="1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f" Nov 02 20:17:41.883207 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.883184 5399 scope.go:122] "RemoveContainer" containerID="1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f" Nov 02 20:17:41.883601 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:41.883577 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f\": container with ID starting with 1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f not found: ID does not exist" containerID="1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f" Nov 02 20:17:41.883688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:41.883609 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f"} err="failed to get container status \"1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f\": rpc error: code = NotFound desc = could not find container \"1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f\": container with ID starting with 1d1007a68639dd078194acfe1c43fbe98c42418d91a62423d39eabe3ae99a93f not found: ID does not exist" Nov 02 20:17:42.027957 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:42.027919 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz"] Nov 02 20:17:42.078329 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:42.078266 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2ac9784d-7cdf44c8f6-5f4qz"] Nov 02 20:17:43.281718 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.281683 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs"] Nov 02 20:17:43.282233 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.281902 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" containerID="cri-o://abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250" gracePeriod=30 Nov 02 20:17:43.347445 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.347302 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv"] Nov 02 20:17:43.348855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.348820 5399 cpu_manager.go:414] "containerMap: removing container" podUID="eaea9b0b-93e7-4c63-83f4-eb8394c69272" containerName="payload-processing" Nov 02 20:17:43.348855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.348854 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaea9b0b-93e7-4c63-83f4-eb8394c69272" containerName="payload-processing" Nov 02 20:17:43.349093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.348906 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7c044378-c69b-45cb-893c-0d98abf6cb85" containerName="payload-pre-processing" Nov 02 20:17:43.349093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.348914 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c044378-c69b-45cb-893c-0d98abf6cb85" containerName="payload-pre-processing" Nov 02 20:17:43.349093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.348950 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c894a85f-10e2-44d9-993b-c50b43ad9734" containerName="payload-processing" Nov 02 20:17:43.349093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.348958 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c894a85f-10e2-44d9-993b-c50b43ad9734" containerName="payload-processing" Nov 02 20:17:43.349093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.348975 5399 cpu_manager.go:414] "containerMap: removing container" podUID="dccd1cf9-eaa9-4f2c-b018-8346316cdfb8" containerName="payload-processing" Nov 02 20:17:43.349093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.348983 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccd1cf9-eaa9-4f2c-b018-8346316cdfb8" containerName="payload-processing" Nov 02 20:17:43.349093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.349003 5399 cpu_manager.go:414] "containerMap: removing container" podUID="9b0848b0-854e-45bd-85fe-fb952b51a9fc" containerName="maas-api" Nov 02 20:17:43.349093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.349010 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0848b0-854e-45bd-85fe-fb952b51a9fc" containerName="maas-api" Nov 02 20:17:43.349093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.349086 5399 cpu_manager.go:414] "containerMap: removing container" podUID="a550c407-5174-4765-a7ec-6b1aeae2080b" containerName="maas-api" Nov 02 20:17:43.349093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.349095 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a550c407-5174-4765-a7ec-6b1aeae2080b" containerName="maas-api" Nov 02 20:17:43.349833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.349523 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a550c407-5174-4765-a7ec-6b1aeae2080b" containerName="maas-api" Nov 02 20:17:43.349833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.349548 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaea9b0b-93e7-4c63-83f4-eb8394c69272" containerName="payload-processing" Nov 02 20:17:43.349833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.349570 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c044378-c69b-45cb-893c-0d98abf6cb85" containerName="payload-pre-processing" Nov 02 20:17:43.349833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.349592 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c894a85f-10e2-44d9-993b-c50b43ad9734" containerName="payload-processing" Nov 02 20:17:43.349833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.349602 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccd1cf9-eaa9-4f2c-b018-8346316cdfb8" containerName="payload-processing" Nov 02 20:17:43.349833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.349619 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0848b0-854e-45bd-85fe-fb952b51a9fc" containerName="maas-api" Nov 02 20:17:43.355784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.355755 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.408415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.407592 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-13c1d955-openshift-default-dockercfg-xqcnw\"" Nov 02 20:17:43.438685 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.438639 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-token\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.438879 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.438703 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xsss\" (UniqueName: \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-kube-api-access-6xsss\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.438879 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.438789 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-data\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.438998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.438919 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-credential-socket\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.439060 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.439014 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fec1f1-6034-4946-92b3-45d90dc143d8-istiod-ca-cert\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.439115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.439070 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-envoy\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.439115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.439108 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-certs\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.439213 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.439190 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-podinfo\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.439262 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.439243 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-socket\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.483049 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.458976 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv"] Nov 02 20:17:43.529688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.529647 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a550c407-5174-4765-a7ec-6b1aeae2080b" path="/var/lib/kubelet/pods/a550c407-5174-4765-a7ec-6b1aeae2080b/volumes" Nov 02 20:17:43.540528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.540433 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-credential-socket\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.540716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.540545 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fec1f1-6034-4946-92b3-45d90dc143d8-istiod-ca-cert\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.540716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.540601 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-envoy\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.540716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.540639 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-certs\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.540901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.540719 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-podinfo\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.540901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.540771 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-socket\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.540901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.540836 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-token\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.540901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.540867 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xsss\" (UniqueName: \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-kube-api-access-6xsss\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.541090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.540913 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-credential-socket\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.541090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.540955 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-data\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.541558 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.541531 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-data\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.541676 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.541620 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-certs\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.542254 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.541671 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fec1f1-6034-4946-92b3-45d90dc143d8-istiod-ca-cert\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.542254 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.541686 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-socket\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.543797 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.543753 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-envoy\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.543917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.543828 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-podinfo\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.594454 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.594416 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-token\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.595938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.595911 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xsss\" (UniqueName: \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-kube-api-access-6xsss\") pod \"e2e-ait-13c1d955-openshift-default-689646446c-q87fv\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:43.757949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:43.757906 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:44.203682 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:17:44.203649 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fec1f1_6034_4946_92b3_45d90dc143d8.slice/crio-481160cb027d790c1641a6f5037b7533ea80f1480fa4b10347c78d0f06cb4ba3 WatchSource:0}: Error finding container 481160cb027d790c1641a6f5037b7533ea80f1480fa4b10347c78d0f06cb4ba3: Status 404 returned error can't find the container with id 481160cb027d790c1641a6f5037b7533ea80f1480fa4b10347c78d0f06cb4ba3 Nov 02 20:17:44.204253 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:44.204232 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv"] Nov 02 20:17:44.206974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:44.206941 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:44.207054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:44.207009 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:44.207054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:44.207049 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:44.932771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:44.932733 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" event={"ID":"56fec1f1-6034-4946-92b3-45d90dc143d8","Type":"ContainerStarted","Data":"25bcc211fe6c441dd174c2e2c7cd8d989b48d4b5fcf7da6d908284dd60adc4eb"} Nov 02 20:17:44.933161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:44.932794 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" event={"ID":"56fec1f1-6034-4946-92b3-45d90dc143d8","Type":"ContainerStarted","Data":"481160cb027d790c1641a6f5037b7533ea80f1480fa4b10347c78d0f06cb4ba3"} Nov 02 20:17:45.133269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:45.133210 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" podStartSLOduration=2.133194957 podStartE2EDuration="2.133194957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:45.086901712 +0000 UTC m=+1507.697553803" watchObservedRunningTime="2026-11-02 20:17:45.133194957 +0000 UTC m=+1507.743847048" Nov 02 20:17:45.758600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:45.758562 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:45.763963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:45.763938 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:45.940449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:45.940423 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:45.941566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:45.941544 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:17:47.712400 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:47.712353 5399 patch_prober.go:29] interesting pod/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:17:47.712794 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:47.712426 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:17:48.239513 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:48.239471 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:17:48.574016 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.573990 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:17:48.632040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632005 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-certs\") pod \"d83953ff-1dd7-4b12-bc48-90ef3658777d\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " Nov 02 20:17:48.632225 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632116 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d83953ff-1dd7-4b12-bc48-90ef3658777d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d83953ff-1dd7-4b12-bc48-90ef3658777d-istiod-ca-cert\") pod \"d83953ff-1dd7-4b12-bc48-90ef3658777d\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " Nov 02 20:17:48.632225 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632160 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-envoy\") pod \"d83953ff-1dd7-4b12-bc48-90ef3658777d\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " Nov 02 20:17:48.632225 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632199 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-podinfo\") pod \"d83953ff-1dd7-4b12-bc48-90ef3658777d\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " Nov 02 20:17:48.632940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632269 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-token\" (UniqueName: \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-token\") pod \"d83953ff-1dd7-4b12-bc48-90ef3658777d\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " Nov 02 20:17:48.632940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632409 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-credential-socket\") pod \"d83953ff-1dd7-4b12-bc48-90ef3658777d\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " Nov 02 20:17:48.632940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632452 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-socket\") pod \"d83953ff-1dd7-4b12-bc48-90ef3658777d\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " Nov 02 20:17:48.632940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632486 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-certs" pod "d83953ff-1dd7-4b12-bc48-90ef3658777d" (UID: "d83953ff-1dd7-4b12-bc48-90ef3658777d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:48.632940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632625 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-data\") pod \"d83953ff-1dd7-4b12-bc48-90ef3658777d\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " Nov 02 20:17:48.632940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632679 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-credential-socket" pod "d83953ff-1dd7-4b12-bc48-90ef3658777d" (UID: "d83953ff-1dd7-4b12-bc48-90ef3658777d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:48.632940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632688 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-socket" pod "d83953ff-1dd7-4b12-bc48-90ef3658777d" (UID: "d83953ff-1dd7-4b12-bc48-90ef3658777d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:48.632940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632701 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83953ff-1dd7-4b12-bc48-90ef3658777d-istiod-ca-cert" pod "d83953ff-1dd7-4b12-bc48-90ef3658777d" (UID: "d83953ff-1dd7-4b12-bc48-90ef3658777d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:48.632940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632723 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-kube-api-access-vbr2b\" (UniqueName: \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-kube-api-access-vbr2b\") pod \"d83953ff-1dd7-4b12-bc48-90ef3658777d\" (UID: \"d83953ff-1dd7-4b12-bc48-90ef3658777d\") " Nov 02 20:17:48.632940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.632890 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-data" pod "d83953ff-1dd7-4b12-bc48-90ef3658777d" (UID: "d83953ff-1dd7-4b12-bc48-90ef3658777d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:48.633747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.633722 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:48.633909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.633752 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:48.633909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.633768 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:48.633909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.633799 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d83953ff-1dd7-4b12-bc48-90ef3658777d-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:48.634091 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.633931 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:48.635085 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.635057 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-token" pod "d83953ff-1dd7-4b12-bc48-90ef3658777d" (UID: "d83953ff-1dd7-4b12-bc48-90ef3658777d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:48.635394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.635365 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-kube-api-access-vbr2b" pod "d83953ff-1dd7-4b12-bc48-90ef3658777d" (UID: "d83953ff-1dd7-4b12-bc48-90ef3658777d"). InnerVolumeSpecName "kube-api-access-vbr2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:48.635492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.635407 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-envoy" pod "d83953ff-1dd7-4b12-bc48-90ef3658777d" (UID: "d83953ff-1dd7-4b12-bc48-90ef3658777d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:17:48.635558 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.635503 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-podinfo" pod "d83953ff-1dd7-4b12-bc48-90ef3658777d" (UID: "d83953ff-1dd7-4b12-bc48-90ef3658777d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:17:48.735337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.735267 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:48.735337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.735333 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:48.735337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.735343 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:48.735850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.735352 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbr2b\" (UniqueName: \"kubernetes.io/projected/d83953ff-1dd7-4b12-bc48-90ef3658777d-kube-api-access-vbr2b\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:48.967082 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.967046 5399 generic.go:356] "Generic (PLEG): container finished" podID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerID="abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250" exitCode=0 Nov 02 20:17:48.967269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.967087 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" event={"ID":"d83953ff-1dd7-4b12-bc48-90ef3658777d","Type":"ContainerDied","Data":"abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250"} Nov 02 20:17:48.967269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.967100 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" Nov 02 20:17:48.967269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.967136 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs" event={"ID":"d83953ff-1dd7-4b12-bc48-90ef3658777d","Type":"ContainerDied","Data":"687a0cdcd23f12859660cfc1c83833cea46e69c340d4337a2a5fc0f239f12b7a"} Nov 02 20:17:48.967269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.967161 5399 scope.go:122] "RemoveContainer" containerID="abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250" Nov 02 20:17:48.978568 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.978548 5399 scope.go:122] "RemoveContainer" containerID="56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92" Nov 02 20:17:48.988448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.988423 5399 scope.go:122] "RemoveContainer" containerID="abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250" Nov 02 20:17:48.988736 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:48.988716 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250\": container with ID starting with abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250 not found: ID does not exist" containerID="abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250" Nov 02 20:17:48.988784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.988746 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250"} err="failed to get container status \"abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250\": rpc error: code = NotFound desc = could not find container \"abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250\": container with ID starting with abe31f729651724dad79d0ef3e5616ca155b6ea794cb976503cb46bee7a84250 not found: ID does not exist" Nov 02 20:17:48.988784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.988764 5399 scope.go:122] "RemoveContainer" containerID="56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92" Nov 02 20:17:48.988991 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:48.988975 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92\": container with ID starting with 56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92 not found: ID does not exist" containerID="56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92" Nov 02 20:17:48.989042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:48.988997 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92"} err="failed to get container status \"56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92\": rpc error: code = NotFound desc = could not find container \"56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92\": container with ID starting with 56404e2f03fb553b3da151c7a171568cbce20a71818569cc45b009dda17bbc92 not found: ID does not exist" Nov 02 20:17:49.122370 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:49.122335 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs"] Nov 02 20:17:49.126391 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:49.126367 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2ac9784d-gw-openshift-default-549f5d6d77-wvhvs"] Nov 02 20:17:49.509417 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:49.509387 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" path="/var/lib/kubelet/pods/d83953ff-1dd7-4b12-bc48-90ef3658777d/volumes" Nov 02 20:17:50.434717 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.434682 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw"] Nov 02 20:17:50.435730 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.435704 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" Nov 02 20:17:50.435730 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.435728 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" Nov 02 20:17:50.435938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.435790 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" Nov 02 20:17:50.435938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.435799 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" Nov 02 20:17:50.436100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.435954 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" Nov 02 20:17:50.436100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.435977 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83953ff-1dd7-4b12-bc48-90ef3658777d" containerName="istio-proxy" Nov 02 20:17:50.443589 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.443564 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.497545 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.485691 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4"] Nov 02 20:17:50.497545 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.491261 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.528034 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.528005 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw"] Nov 02 20:17:50.569367 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.569320 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.569772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.569729 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.570005 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.569968 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.570206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.570185 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.570394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.570373 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhf8n\" (UniqueName: \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-kube-api-access-bhf8n\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.570564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.570548 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.570773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.570755 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2137fc-c108-477f-b6ee-8d36b7e63189-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.570915 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.570899 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.571031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.571017 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.571219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.571203 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.575093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.571506 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.575093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.571547 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm2f8\" (UniqueName: \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-kube-api-access-mm2f8\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.577573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.577537 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4"] Nov 02 20:17:50.578037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.578011 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.578128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.578059 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.578252 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.578234 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.578373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.578271 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.578433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.578390 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.578496 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.578468 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.680456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.680418 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.680649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.680513 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.680649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.680582 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.680770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.680650 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2f8\" (UniqueName: \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-kube-api-access-mm2f8\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.680770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.680737 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.680869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.680781 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.680869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.680858 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.680958 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.680894 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.681011 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.680962 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.681058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.681016 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.681762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.681152 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.681762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.681239 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.681762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.681318 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.681762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.681367 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.681762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.681415 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhf8n\" (UniqueName: \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-kube-api-access-bhf8n\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.681762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.681460 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.681762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.681542 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2137fc-c108-477f-b6ee-8d36b7e63189-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.681762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.681589 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.686337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.683075 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.686337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.683395 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.687146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.686805 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.687146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.686822 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.695614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.695575 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.695917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.695888 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.696268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.696244 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2137fc-c108-477f-b6ee-8d36b7e63189-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.696559 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.696536 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.698402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.698376 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.701932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.700584 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.702600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.702578 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.702890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.702850 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.704942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.704898 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.705429 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.705402 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.737245 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.737171 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.741190 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.741157 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2f8\" (UniqueName: \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-kube-api-access-mm2f8\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.749842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.749779 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-qqdvw\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.749966 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.749897 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhf8n\" (UniqueName: \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-kube-api-access-bhf8n\") pod \"maas-default-gateway-openshift-default-696f47c6c-jg6q4\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:50.784069 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.783445 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:50.833066 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:50.833033 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:51.232389 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:17:51.232349 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2137fc_c108_477f_b6ee_8d36b7e63189.slice/crio-95e35395b6e98916c67a3701eec2798d3588202079076d8274d9118d21fe1bf7 WatchSource:0}: Error finding container 95e35395b6e98916c67a3701eec2798d3588202079076d8274d9118d21fe1bf7: Status 404 returned error can't find the container with id 95e35395b6e98916c67a3701eec2798d3588202079076d8274d9118d21fe1bf7 Nov 02 20:17:51.233602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.233408 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw"] Nov 02 20:17:51.235969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.235929 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:51.236050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.236016 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:51.236084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.236054 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:51.284257 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.284230 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4"] Nov 02 20:17:51.291505 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:17:51.291473 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ed5d06_4f0e_4264_9219_bf19ab89dbff.slice/crio-fc81d1d7154b7b00906f56b50fbab70f6bfc1a7c8179737862820d13074d9f9d WatchSource:0}: Error finding container fc81d1d7154b7b00906f56b50fbab70f6bfc1a7c8179737862820d13074d9f9d: Status 404 returned error can't find the container with id fc81d1d7154b7b00906f56b50fbab70f6bfc1a7c8179737862820d13074d9f9d Nov 02 20:17:51.294887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.294854 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:51.294998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.294944 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:51.294998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.294974 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:51.996048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.996011 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" event={"ID":"39ed5d06-4f0e-4264-9219-bf19ab89dbff","Type":"ContainerStarted","Data":"f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf"} Nov 02 20:17:51.996560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.996072 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" event={"ID":"39ed5d06-4f0e-4264-9219-bf19ab89dbff","Type":"ContainerStarted","Data":"fc81d1d7154b7b00906f56b50fbab70f6bfc1a7c8179737862820d13074d9f9d"} Nov 02 20:17:51.997709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.997678 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" event={"ID":"aa2137fc-c108-477f-b6ee-8d36b7e63189","Type":"ContainerStarted","Data":"345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506"} Nov 02 20:17:51.997822 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:51.997724 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" event={"ID":"aa2137fc-c108-477f-b6ee-8d36b7e63189","Type":"ContainerStarted","Data":"95e35395b6e98916c67a3701eec2798d3588202079076d8274d9118d21fe1bf7"} Nov 02 20:17:52.119626 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:52.119572 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podStartSLOduration=2.119554496 podStartE2EDuration="2.119554496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:52.115481711 +0000 UTC m=+1514.726133802" watchObservedRunningTime="2026-11-02 20:17:52.119554496 +0000 UTC m=+1514.730206648" Nov 02 20:17:52.253399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:52.247506 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podStartSLOduration=2.247486409 podStartE2EDuration="2.247486409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:52.245790168 +0000 UTC m=+1514.856442286" watchObservedRunningTime="2026-11-02 20:17:52.247486409 +0000 UTC m=+1514.858138501" Nov 02 20:17:52.784250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:52.784222 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:17:52.785657 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:52.785629 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qqdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:17:52.785804 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:52.785687 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:17:52.847923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:52.847889 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:17:52.868300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:52.855216 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jg6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 20:17:52.868300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:52.855298 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 20:17:53.784095 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:53.784059 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qqdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:17:53.784578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:53.784121 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:17:53.835745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:53.835702 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jg6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 20:17:53.836060 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:53.836028 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 20:17:54.791852 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:54.791811 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qqdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:17:54.793875 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:54.793835 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:17:54.836305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:54.835673 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jg6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 20:17:54.836305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:54.835735 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 20:17:55.784857 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:55.784820 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qqdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:17:55.785023 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:55.784872 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:17:55.851300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:55.847753 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jg6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 20:17:55.851300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:55.847821 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 20:17:56.784341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:56.784302 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qqdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:17:56.784504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:56.784357 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:17:56.844303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:56.840940 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jg6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 20:17:56.844303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:56.840995 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 20:17:57.009769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.009732 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp"] Nov 02 20:17:57.010172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.009956 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" podUID="92e5d4f1-2e2b-44f2-8732-cf8fec071edb" containerName="maas-api" containerID="cri-o://7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2" gracePeriod=30 Nov 02 20:17:57.095138 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:57.094720 5399 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e5d4f1_2e2b_44f2_8732_cf8fec071edb.slice/crio-7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:17:57.647878 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.647848 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:17:57.830308 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.815614 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qqdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:17:57.830308 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.815679 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:17:57.832497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.830816 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-metrics-certs\") pod \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " Nov 02 20:17:57.832497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.831192 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls\") pod \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " Nov 02 20:17:57.832497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.831287 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-openshift-service-ca\") pod \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " Nov 02 20:17:57.832497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.831346 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-kube-api-access-p9vbk\" (UniqueName: \"kubernetes.io/projected/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-kube-api-access-p9vbk\") pod \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\" (UID: \"92e5d4f1-2e2b-44f2-8732-cf8fec071edb\") " Nov 02 20:17:57.835907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.833119 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-openshift-service-ca" pod "92e5d4f1-2e2b-44f2-8732-cf8fec071edb" (UID: "92e5d4f1-2e2b-44f2-8732-cf8fec071edb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:17:57.835907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.833542 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jg6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 20:17:57.835907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.833603 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 20:17:57.838141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.838109 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-metrics-certs" pod "92e5d4f1-2e2b-44f2-8732-cf8fec071edb" (UID: "92e5d4f1-2e2b-44f2-8732-cf8fec071edb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:57.838350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.838132 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls" pod "92e5d4f1-2e2b-44f2-8732-cf8fec071edb" (UID: "92e5d4f1-2e2b-44f2-8732-cf8fec071edb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:17:57.843565 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.843529 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-kube-api-access-p9vbk" pod "92e5d4f1-2e2b-44f2-8732-cf8fec071edb" (UID: "92e5d4f1-2e2b-44f2-8732-cf8fec071edb"). InnerVolumeSpecName "kube-api-access-p9vbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:17:57.933833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.933785 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:57.933833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.933831 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:57.934097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.933848 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9vbk\" (UniqueName: \"kubernetes.io/projected/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-kube-api-access-p9vbk\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:57.934097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:57.933864 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e5d4f1-2e2b-44f2-8732-cf8fec071edb-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:17:58.120462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.120367 5399 generic.go:356] "Generic (PLEG): container finished" podID="92e5d4f1-2e2b-44f2-8732-cf8fec071edb" containerID="7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2" exitCode=0 Nov 02 20:17:58.121061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.120433 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" Nov 02 20:17:58.121321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.120457 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" event={"ID":"92e5d4f1-2e2b-44f2-8732-cf8fec071edb","Type":"ContainerDied","Data":"7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2"} Nov 02 20:17:58.121418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.121398 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp" event={"ID":"92e5d4f1-2e2b-44f2-8732-cf8fec071edb","Type":"ContainerDied","Data":"c446546e4c3be64e65d547fb71e6d8acf684cf80c059a006e80662a8895c7f0f"} Nov 02 20:17:58.121482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.121438 5399 scope.go:122] "RemoveContainer" containerID="7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2" Nov 02 20:17:58.241700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.241465 5399 scope.go:122] "RemoveContainer" containerID="7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2" Nov 02 20:17:58.241835 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:17:58.241785 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2\": container with ID starting with 7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2 not found: ID does not exist" containerID="7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2" Nov 02 20:17:58.241835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.241819 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2"} err="failed to get container status \"7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2\": rpc error: code = NotFound desc = could not find container \"7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2\": container with ID starting with 7a1274745f3bf369dd3704931855e6673871ede314e8bbf75089578397c3dfe2 not found: ID does not exist" Nov 02 20:17:58.354341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.354185 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp"] Nov 02 20:17:58.371121 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.371034 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d73d56c-79966df6cb-fsbjp"] Nov 02 20:17:58.784779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.784679 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qqdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:17:58.784987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.784793 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:17:58.834168 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.834122 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jg6q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 20:17:58.834391 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:58.834193 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 20:17:59.511362 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:17:59.510713 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e5d4f1-2e2b-44f2-8732-cf8fec071edb" path="/var/lib/kubelet/pods/92e5d4f1-2e2b-44f2-8732-cf8fec071edb/volumes" Nov 02 20:18:00.475799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:00.475761 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:18:00.477491 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:00.477460 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:18:00.478448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:00.478428 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:18:00.582228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:00.582193 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:18:00.582771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:00.582636 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:18:00.583876 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:00.583852 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:18:01.466746 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.466711 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx"] Nov 02 20:18:01.467422 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.467408 5399 cpu_manager.go:414] "containerMap: removing container" podUID="92e5d4f1-2e2b-44f2-8732-cf8fec071edb" containerName="maas-api" Nov 02 20:18:01.467471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.467423 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e5d4f1-2e2b-44f2-8732-cf8fec071edb" containerName="maas-api" Nov 02 20:18:01.467573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.467564 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e5d4f1-2e2b-44f2-8732-cf8fec071edb" containerName="maas-api" Nov 02 20:18:01.472661 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.472638 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.530796 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.530759 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx"] Nov 02 20:18:01.541813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.541772 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-be1348-openshift-default-dockercfg-w8nbm\"" Nov 02 20:18:01.547908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.547869 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-certs\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.548086 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.548047 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-envoy\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.548161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.548115 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd261a1-a473-4c79-a8d8-71d19923c944-istiod-ca-cert\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.548346 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.548266 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-data\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.548461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.548399 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-socket\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.548641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.548617 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-token\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.548808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.548790 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-credential-socket\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.548960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.548937 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgsnx\" (UniqueName: \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-kube-api-access-vgsnx\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.549084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.549065 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-podinfo\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.650808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.650769 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-data\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.650863 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-socket\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.650895 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-token\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.650984 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-credential-socket\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.651049 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgsnx\" (UniqueName: \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-kube-api-access-vgsnx\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.651085 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-podinfo\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.651111 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-data\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.651182 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-socket\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.651193 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-certs\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.651245 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-envoy\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651696 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.651328 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd261a1-a473-4c79-a8d8-71d19923c944-istiod-ca-cert\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651696 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.651388 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-credential-socket\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651696 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.651432 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-certs\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.651964 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.651947 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd261a1-a473-4c79-a8d8-71d19923c944-istiod-ca-cert\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.653455 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.653431 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-envoy\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.653651 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.653634 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-podinfo\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.740441 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.740323 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgsnx\" (UniqueName: \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-kube-api-access-vgsnx\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.743653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.743615 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-token\") pod \"e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:01.788895 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:01.788849 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:02.105795 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.105729 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc"] Nov 02 20:18:02.123327 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.118262 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:18:02.170299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.170235 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc"] Nov 02 20:18:02.173571 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.173533 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-plugins-config-volume\") pod \"payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc\" (UID: \"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80\") " pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:18:02.173905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.173877 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k49gj\" (UniqueName: \"kubernetes.io/projected/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-kube-api-access-k49gj\") pod \"payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc\" (UID: \"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80\") " pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:18:02.175707 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.175682 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-55e81986\"" Nov 02 20:18:02.175869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.175725 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-55e81986-dockercfg-t2hht\"" Nov 02 20:18:02.284946 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.284906 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k49gj\" (UniqueName: \"kubernetes.io/projected/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-kube-api-access-k49gj\") pod \"payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc\" (UID: \"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80\") " pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:18:02.290934 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.290886 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-plugins-config-volume\") pod \"payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc\" (UID: \"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80\") " pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:18:02.302357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.302317 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-plugins-config-volume\") pod \"payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc\" (UID: \"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80\") " pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:18:02.304808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.304776 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx"] Nov 02 20:18:02.304940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.304882 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:18:02.305009 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.304966 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:18:02.305071 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.305013 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:18:02.339985 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.339954 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k49gj\" (UniqueName: \"kubernetes.io/projected/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-kube-api-access-k49gj\") pod \"payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc\" (UID: \"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80\") " pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:18:02.448334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.446643 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:18:02.890754 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:18:02.890719 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1637a4_b052_43a4_be4d_f4e4f0d2ae80.slice/crio-161da23823bed43dc0b14437693b08e86586c450d7dad6bd3f210d8c6471765a WatchSource:0}: Error finding container 161da23823bed43dc0b14437693b08e86586c450d7dad6bd3f210d8c6471765a: Status 404 returned error can't find the container with id 161da23823bed43dc0b14437693b08e86586c450d7dad6bd3f210d8c6471765a Nov 02 20:18:02.891449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:02.891425 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc"] Nov 02 20:18:03.182954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:03.182850 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" event={"ID":"5fd261a1-a473-4c79-a8d8-71d19923c944","Type":"ContainerStarted","Data":"a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a"} Nov 02 20:18:03.182954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:03.182922 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" event={"ID":"5fd261a1-a473-4c79-a8d8-71d19923c944","Type":"ContainerStarted","Data":"63feb3fff5d50e33111d5ef37684d4084d57e1ce9bbb6f5bc4131fe7bc339f54"} Nov 02 20:18:03.184759 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:03.184730 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" event={"ID":"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80","Type":"ContainerStarted","Data":"2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9"} Nov 02 20:18:03.184900 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:03.184779 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" event={"ID":"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80","Type":"ContainerStarted","Data":"161da23823bed43dc0b14437693b08e86586c450d7dad6bd3f210d8c6471765a"} Nov 02 20:18:03.184900 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:03.184868 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:18:03.296207 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:03.296141 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" podStartSLOduration=2.296120654 podStartE2EDuration="2.296120654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:03.292014548 +0000 UTC m=+1525.902666640" watchObservedRunningTime="2026-11-02 20:18:03.296120654 +0000 UTC m=+1525.906772747" Nov 02 20:18:03.398843 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:03.398786 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" podStartSLOduration=1.3987706 podStartE2EDuration="1.3987706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:03.397481056 +0000 UTC m=+1526.008133147" watchObservedRunningTime="2026-11-02 20:18:03.3987706 +0000 UTC m=+1526.009422691" Nov 02 20:18:03.789755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:03.789723 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:03.795526 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:03.795498 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:04.192438 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:04.192410 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:04.193358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:04.193334 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:18:09.768349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:09.768312 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww"] Nov 02 20:18:09.773402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:09.773376 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:18:09.844590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:09.844546 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66pt\" (UniqueName: \"kubernetes.io/projected/3cc9167c-b871-405c-b8e9-fc901deb83b7-kube-api-access-h66pt\") pod \"payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww\" (UID: \"3cc9167c-b871-405c-b8e9-fc901deb83b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:18:09.844859 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:09.844833 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc9167c-b871-405c-b8e9-fc901deb83b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww\" (UID: \"3cc9167c-b871-405c-b8e9-fc901deb83b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:18:09.878578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:09.878539 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-be1348\"" Nov 02 20:18:09.878578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:09.878539 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-be1348-dockercfg-lf42z\"" Nov 02 20:18:09.886618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:09.886591 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww"] Nov 02 20:18:09.948716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:09.948666 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66pt\" (UniqueName: \"kubernetes.io/projected/3cc9167c-b871-405c-b8e9-fc901deb83b7-kube-api-access-h66pt\") pod \"payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww\" (UID: \"3cc9167c-b871-405c-b8e9-fc901deb83b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:18:09.948945 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:09.948922 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc9167c-b871-405c-b8e9-fc901deb83b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww\" (UID: \"3cc9167c-b871-405c-b8e9-fc901deb83b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:18:09.955331 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:09.949989 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc9167c-b871-405c-b8e9-fc901deb83b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww\" (UID: \"3cc9167c-b871-405c-b8e9-fc901deb83b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:18:10.007019 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:10.006983 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66pt\" (UniqueName: \"kubernetes.io/projected/3cc9167c-b871-405c-b8e9-fc901deb83b7-kube-api-access-h66pt\") pod \"payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww\" (UID: \"3cc9167c-b871-405c-b8e9-fc901deb83b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:18:10.088420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:10.088383 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:18:10.503837 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:10.503813 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww"] Nov 02 20:18:10.514804 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:18:10.514461 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc9167c_b871_405c_b8e9_fc901deb83b7.slice/crio-280cfbb662bf0a32a13b1db8f405e06348090bfe40c28e928dfd1ce1b9809a6f WatchSource:0}: Error finding container 280cfbb662bf0a32a13b1db8f405e06348090bfe40c28e928dfd1ce1b9809a6f: Status 404 returned error can't find the container with id 280cfbb662bf0a32a13b1db8f405e06348090bfe40c28e928dfd1ce1b9809a6f Nov 02 20:18:11.269544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:11.269503 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" event={"ID":"3cc9167c-b871-405c-b8e9-fc901deb83b7","Type":"ContainerStarted","Data":"77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269"} Nov 02 20:18:11.269969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:11.269607 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" event={"ID":"3cc9167c-b871-405c-b8e9-fc901deb83b7","Type":"ContainerStarted","Data":"280cfbb662bf0a32a13b1db8f405e06348090bfe40c28e928dfd1ce1b9809a6f"} Nov 02 20:18:11.269969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:11.269744 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:18:11.375631 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:11.375572 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" podStartSLOduration=2.375556885 podStartE2EDuration="2.375556885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:11.373162852 +0000 UTC m=+1533.983814943" watchObservedRunningTime="2026-11-02 20:18:11.375556885 +0000 UTC m=+1533.986208976" Nov 02 20:18:14.195607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:14.195579 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:18:18.238684 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:18:18.238604 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:18:18.846662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:18.846625 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq"] Nov 02 20:18:18.846896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:18.846872 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" containerID="cri-o://69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9" gracePeriod=30 Nov 02 20:18:19.367897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:19.367850 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:18:22.281820 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:22.281791 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:18:24.125287 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.125213 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:18:24.186956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.186921 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-envoy\") pod \"d2853491-636a-4980-9fe1-a90a3a06a5f0\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " Nov 02 20:18:24.187133 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187002 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-certs\") pod \"d2853491-636a-4980-9fe1-a90a3a06a5f0\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " Nov 02 20:18:24.187133 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187067 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-credential-socket\") pod \"d2853491-636a-4980-9fe1-a90a3a06a5f0\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " Nov 02 20:18:24.187133 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187116 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-podinfo\") pod \"d2853491-636a-4980-9fe1-a90a3a06a5f0\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " Nov 02 20:18:24.187380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187139 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-data\") pod \"d2853491-636a-4980-9fe1-a90a3a06a5f0\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " Nov 02 20:18:24.187380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187310 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-certs" pod "d2853491-636a-4980-9fe1-a90a3a06a5f0" (UID: "d2853491-636a-4980-9fe1-a90a3a06a5f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:24.187380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187318 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-credential-socket" pod "d2853491-636a-4980-9fe1-a90a3a06a5f0" (UID: "d2853491-636a-4980-9fe1-a90a3a06a5f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:24.187503 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187386 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-socket\") pod \"d2853491-636a-4980-9fe1-a90a3a06a5f0\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " Nov 02 20:18:24.187503 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187440 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2853491-636a-4980-9fe1-a90a3a06a5f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2853491-636a-4980-9fe1-a90a3a06a5f0-istiod-ca-cert\") pod \"d2853491-636a-4980-9fe1-a90a3a06a5f0\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " Nov 02 20:18:24.187613 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187499 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-data" pod "d2853491-636a-4980-9fe1-a90a3a06a5f0" (UID: "d2853491-636a-4980-9fe1-a90a3a06a5f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:24.187613 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187560 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-kube-api-access-4m4lt\" (UniqueName: \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-kube-api-access-4m4lt\") pod \"d2853491-636a-4980-9fe1-a90a3a06a5f0\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " Nov 02 20:18:24.187613 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187596 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-token\" (UniqueName: \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-token\") pod \"d2853491-636a-4980-9fe1-a90a3a06a5f0\" (UID: \"d2853491-636a-4980-9fe1-a90a3a06a5f0\") " Nov 02 20:18:24.187787 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187657 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-socket" pod "d2853491-636a-4980-9fe1-a90a3a06a5f0" (UID: "d2853491-636a-4980-9fe1-a90a3a06a5f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:24.187895 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.187861 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2853491-636a-4980-9fe1-a90a3a06a5f0-istiod-ca-cert" pod "d2853491-636a-4980-9fe1-a90a3a06a5f0" (UID: "d2853491-636a-4980-9fe1-a90a3a06a5f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:24.188339 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.188317 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:24.188444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.188342 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:24.188444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.188356 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:24.188444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.188368 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:24.188444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.188381 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2853491-636a-4980-9fe1-a90a3a06a5f0-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:24.189401 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.189377 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-envoy" pod "d2853491-636a-4980-9fe1-a90a3a06a5f0" (UID: "d2853491-636a-4980-9fe1-a90a3a06a5f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:24.189659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.189633 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-token" pod "d2853491-636a-4980-9fe1-a90a3a06a5f0" (UID: "d2853491-636a-4980-9fe1-a90a3a06a5f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:24.189756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.189704 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-podinfo" pod "d2853491-636a-4980-9fe1-a90a3a06a5f0" (UID: "d2853491-636a-4980-9fe1-a90a3a06a5f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:18:24.189756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.189738 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-kube-api-access-4m4lt" pod "d2853491-636a-4980-9fe1-a90a3a06a5f0" (UID: "d2853491-636a-4980-9fe1-a90a3a06a5f0"). InnerVolumeSpecName "kube-api-access-4m4lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:24.289378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.289348 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:24.289378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.289375 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m4lt\" (UniqueName: \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-kube-api-access-4m4lt\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:24.289378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.289386 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:24.289600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.289395 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2853491-636a-4980-9fe1-a90a3a06a5f0-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:24.368589 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.368549 5399 generic.go:356] "Generic (PLEG): container finished" podID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerID="69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9" exitCode=0 Nov 02 20:18:24.368761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.368595 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" Nov 02 20:18:24.368761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.368636 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" event={"ID":"d2853491-636a-4980-9fe1-a90a3a06a5f0","Type":"ContainerDied","Data":"69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9"} Nov 02 20:18:24.368761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.368721 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq" event={"ID":"d2853491-636a-4980-9fe1-a90a3a06a5f0","Type":"ContainerDied","Data":"54f3fa91e3226dd63015cb0c94a14e4b537f15cb46a20f6233f9f39654219a30"} Nov 02 20:18:24.368761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.368745 5399 scope.go:122] "RemoveContainer" containerID="69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9" Nov 02 20:18:24.383227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.383199 5399 scope.go:122] "RemoveContainer" containerID="69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9" Nov 02 20:18:24.383583 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:18:24.383559 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9\": container with ID starting with 69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9 not found: ID does not exist" containerID="69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9" Nov 02 20:18:24.383669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.383594 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9"} err="failed to get container status \"69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9\": rpc error: code = NotFound desc = could not find container \"69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9\": container with ID starting with 69323dd9722fa934653d76293549e3c448df74b7455d80c91358b491524360b9 not found: ID does not exist" Nov 02 20:18:24.524876 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.524839 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq"] Nov 02 20:18:24.535207 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:24.535168 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-7qxdq"] Nov 02 20:18:25.511483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:25.511452 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" path="/var/lib/kubelet/pods/d2853491-636a-4980-9fe1-a90a3a06a5f0/volumes" Nov 02 20:18:27.907619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.907578 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5"] Nov 02 20:18:27.908340 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.908323 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" Nov 02 20:18:27.908395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.908342 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" Nov 02 20:18:27.908476 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.908467 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2853491-636a-4980-9fe1-a90a3a06a5f0" containerName="istio-proxy" Nov 02 20:18:27.912138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.912122 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:27.944714 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.944680 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:27.944874 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.944724 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:27.944874 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.944751 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:27.944961 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.944924 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:27.945039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.945025 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:27.945187 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.945174 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:27.945339 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.945324 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvfxs\" (UniqueName: \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-kube-api-access-pvfxs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:27.960660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.960629 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 20:18:27.960660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.960646 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 20:18:27.960850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.960629 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 20:18:27.960887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:27.960864 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-42zng\"" Nov 02 20:18:28.014553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.014518 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5"] Nov 02 20:18:28.046879 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.046840 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.047056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.047039 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.047198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.047181 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvfxs\" (UniqueName: \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-kube-api-access-pvfxs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.047269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.047252 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.047353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.047308 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.047353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.047346 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.047450 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.047421 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.048255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.048224 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.049527 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.049503 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.049694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.049672 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.050112 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.050087 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.050387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.050154 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.105502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.105474 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.105819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.105794 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvfxs\" (UniqueName: \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-kube-api-access-pvfxs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9rlz5\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.225997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.225918 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:28.648963 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:18:28.648933 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ac9c5f_54f3_4ba8_911c_ebe5d560516f.slice/crio-26c4d689f7166d59d0494ea188a6aaece863dc0e897feedb772d63ecc456432b WatchSource:0}: Error finding container 26c4d689f7166d59d0494ea188a6aaece863dc0e897feedb772d63ecc456432b: Status 404 returned error can't find the container with id 26c4d689f7166d59d0494ea188a6aaece863dc0e897feedb772d63ecc456432b Nov 02 20:18:28.649365 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:28.649307 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5"] Nov 02 20:18:29.413160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:29.413127 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" event={"ID":"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f","Type":"ContainerStarted","Data":"26c4d689f7166d59d0494ea188a6aaece863dc0e897feedb772d63ecc456432b"} Nov 02 20:18:31.805389 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:31.805351 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv"] Nov 02 20:18:31.813983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:31.813955 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:31.874410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:31.874378 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv"] Nov 02 20:18:31.910787 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:31.910750 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-metrics-certs\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:31.911062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:31.911035 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-maas-api-tls\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:31.911194 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:31.911166 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqpzc\" (UniqueName: \"kubernetes.io/projected/d6dd6445-0aab-421f-ad16-0378b915f725-kube-api-access-lqpzc\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:31.911270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:31.911226 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dd6445-0aab-421f-ad16-0378b915f725-openshift-service-ca\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:31.914623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:31.914595 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-13c1d955-serving-cert\"" Nov 02 20:18:32.012518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.012486 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-metrics-certs\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:32.012697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.012661 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-maas-api-tls\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:32.012808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.012719 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqpzc\" (UniqueName: \"kubernetes.io/projected/d6dd6445-0aab-421f-ad16-0378b915f725-kube-api-access-lqpzc\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:32.012808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.012745 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dd6445-0aab-421f-ad16-0378b915f725-openshift-service-ca\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:32.013593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.013563 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dd6445-0aab-421f-ad16-0378b915f725-openshift-service-ca\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:32.015221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.015190 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-metrics-certs\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:32.015339 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.015246 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-maas-api-tls\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:32.068162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.068133 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqpzc\" (UniqueName: \"kubernetes.io/projected/d6dd6445-0aab-421f-ad16-0378b915f725-kube-api-access-lqpzc\") pod \"maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:32.131528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.131495 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:32.533421 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:18:32.533389 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dd6445_0aab_421f_ad16_0378b915f725.slice/crio-65afe7cf6545ec0da8bb4d77aaf572dc4812ff5ac77f3ccdce87d4aaf0f41d20 WatchSource:0}: Error finding container 65afe7cf6545ec0da8bb4d77aaf572dc4812ff5ac77f3ccdce87d4aaf0f41d20: Status 404 returned error can't find the container with id 65afe7cf6545ec0da8bb4d77aaf572dc4812ff5ac77f3ccdce87d4aaf0f41d20 Nov 02 20:18:32.533421 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.533398 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv"] Nov 02 20:18:32.537381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.537348 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:18:32.537465 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:32.537428 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:18:33.447408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:33.447366 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" event={"ID":"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f","Type":"ContainerStarted","Data":"e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f"} Nov 02 20:18:33.447899 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:33.447724 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:33.448914 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:33.448883 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" event={"ID":"d6dd6445-0aab-421f-ad16-0378b915f725","Type":"ContainerStarted","Data":"65afe7cf6545ec0da8bb4d77aaf572dc4812ff5ac77f3ccdce87d4aaf0f41d20"} Nov 02 20:18:33.449980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:33.449955 5399 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9rlz5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:18:33.450089 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:33.450032 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" podUID="a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:18:33.644885 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:33.644822 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" podStartSLOduration=2.75986839 podStartE2EDuration="6.644808015s" totalImagesPullingTime="3.884939625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:33.602990142 +0000 UTC m=+1556.213642234" watchObservedRunningTime="2026-11-02 20:18:33.644808015 +0000 UTC m=+1556.255460106" Nov 02 20:18:34.460270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:34.460229 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" event={"ID":"d6dd6445-0aab-421f-ad16-0378b915f725","Type":"ContainerStarted","Data":"ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543"} Nov 02 20:18:34.460816 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:34.460665 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:34.461343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:34.461320 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:18:34.572471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:34.572418 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" podStartSLOduration=2.631947993 podStartE2EDuration="3.572404847s" totalImagesPullingTime="940.456854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:34.568490401 +0000 UTC m=+1557.179142489" watchObservedRunningTime="2026-11-02 20:18:34.572404847 +0000 UTC m=+1557.183056938" Nov 02 20:18:40.474526 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:40.474495 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:18:45.285660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.285621 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj"] Nov 02 20:18:45.291933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.291910 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.336269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.336231 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b4b611-a84f-4da7-bae2-e26d177a6753-openshift-service-ca\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.336523 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.336495 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-metrics-certs\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.336687 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.336655 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-maas-api-tls\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.336817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.336793 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snmbd\" (UniqueName: \"kubernetes.io/projected/33b4b611-a84f-4da7-bae2-e26d177a6753-kube-api-access-snmbd\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.342318 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.340381 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-55e81986-serving-cert\"" Nov 02 20:18:45.343098 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.343074 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj"] Nov 02 20:18:45.438718 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.438665 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snmbd\" (UniqueName: \"kubernetes.io/projected/33b4b611-a84f-4da7-bae2-e26d177a6753-kube-api-access-snmbd\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.438915 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.438860 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b4b611-a84f-4da7-bae2-e26d177a6753-openshift-service-ca\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.439053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.439031 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-metrics-certs\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.439168 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.439153 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-maas-api-tls\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.439465 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:18:45.439446 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-serving-cert: secret "maas-api-e2e-ait-55e81986-serving-cert" not found Nov 02 20:18:45.439575 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:18:45.439509 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-maas-api-tls podName:33b4b611-a84f-4da7-bae2-e26d177a6753 nodeName:}" failed. No retries permitted until 2026-11-02 20:18:45.939490475 +0000 UTC m=+1568.550142546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-maas-api-tls") pod "maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" (UID: "33b4b611-a84f-4da7-bae2-e26d177a6753") : secret "maas-api-e2e-ait-55e81986-serving-cert" not found Nov 02 20:18:45.439859 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.439833 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b4b611-a84f-4da7-bae2-e26d177a6753-openshift-service-ca\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.442246 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.442219 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-metrics-certs\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.492878 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.492848 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snmbd\" (UniqueName: \"kubernetes.io/projected/33b4b611-a84f-4da7-bae2-e26d177a6753-kube-api-access-snmbd\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.955203 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.955166 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-maas-api-tls\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:45.957748 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:45.957724 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-maas-api-tls\") pod \"maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:46.207010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:46.206923 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:46.582968 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:18:46.582937 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b4b611_a84f_4da7_bae2_e26d177a6753.slice/crio-f2ad804957ff655a072c9d3c9bc94fdf79302e5206fca2e56278f1479c2aa61c WatchSource:0}: Error finding container f2ad804957ff655a072c9d3c9bc94fdf79302e5206fca2e56278f1479c2aa61c: Status 404 returned error can't find the container with id f2ad804957ff655a072c9d3c9bc94fdf79302e5206fca2e56278f1479c2aa61c Nov 02 20:18:46.584325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:46.584299 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj"] Nov 02 20:18:47.578386 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:47.578344 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" event={"ID":"33b4b611-a84f-4da7-bae2-e26d177a6753","Type":"ContainerStarted","Data":"0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1"} Nov 02 20:18:47.578572 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:47.578410 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" event={"ID":"33b4b611-a84f-4da7-bae2-e26d177a6753","Type":"ContainerStarted","Data":"f2ad804957ff655a072c9d3c9bc94fdf79302e5206fca2e56278f1479c2aa61c"} Nov 02 20:18:47.578572 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:47.578438 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:18:47.689404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:47.689303 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" podStartSLOduration=1.877517123 podStartE2EDuration="2.689287738s" totalImagesPullingTime="811.770615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:47.686682498 +0000 UTC m=+1570.297334591" watchObservedRunningTime="2026-11-02 20:18:47.689287738 +0000 UTC m=+1570.299939820" Nov 02 20:18:48.274341 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:18:48.272555 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:18:53.592377 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:18:53.592346 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:19:10.865511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:10.865471 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:19:13.071299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:13.070938 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9"] Nov 02 20:19:13.098625 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:13.098586 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955bcc67-vx5x9"] Nov 02 20:19:13.510816 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:13.510740 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb09c2e-faa8-4231-afcb-6b3e08998ead" path="/var/lib/kubelet/pods/ddb09c2e-faa8-4231-afcb-6b3e08998ead/volumes" Nov 02 20:19:16.590587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:16.586561 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qqdvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:19:16.590587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:16.586647 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:19:16.621678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:16.621641 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks"] Nov 02 20:19:16.632428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:16.632388 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jg6q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:19:16.632707 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:16.632674 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:19:17.100673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:17.100632 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg"] Nov 02 20:19:17.109948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:17.109911 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed60304-2m9sg"] Nov 02 20:19:17.512059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:17.511980 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ab431d-ed05-470d-8475-99df61005fe4" path="/var/lib/kubelet/pods/d2ab431d-ed05-470d-8475-99df61005fe4/volumes" Nov 02 20:19:18.244228 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:19:18.244200 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:19:21.622664 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:21.622608 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" podUID="37aa955b-ef9d-4016-8775-6df4bb3bcd2a" containerName="payload-processing" containerID="cri-o://b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8" gracePeriod=25 Nov 02 20:19:21.912057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:21.912016 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:19:21.991831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:21.991797 5399 generic.go:356] "Generic (PLEG): container finished" podID="37aa955b-ef9d-4016-8775-6df4bb3bcd2a" containerID="b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8" exitCode=0 Nov 02 20:19:21.991995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:21.991854 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" Nov 02 20:19:21.991995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:21.991878 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" event={"ID":"37aa955b-ef9d-4016-8775-6df4bb3bcd2a","Type":"ContainerDied","Data":"b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8"} Nov 02 20:19:21.991995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:21.991935 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks" event={"ID":"37aa955b-ef9d-4016-8775-6df4bb3bcd2a","Type":"ContainerDied","Data":"5fb8e05791d20aaf019d70b6e52b9be33a0dbba865675bdd6ffc41b2eaad5998"} Nov 02 20:19:21.991995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:21.991964 5399 scope.go:122] "RemoveContainer" containerID="b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8" Nov 02 20:19:22.002664 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:22.002644 5399 scope.go:122] "RemoveContainer" containerID="b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8" Nov 02 20:19:22.002963 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:19:22.002938 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8\": container with ID starting with b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8 not found: ID does not exist" containerID="b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8" Nov 02 20:19:22.003020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:22.002976 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8"} err="failed to get container status \"b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8\": rpc error: code = NotFound desc = could not find container \"b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8\": container with ID starting with b2f9b86f48b516b51ad8b2829f116bd03190e4e53e48374d70b2b845c660bbe8 not found: ID does not exist" Nov 02 20:19:22.003262 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:22.003246 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-plugins-config-volume\") pod \"37aa955b-ef9d-4016-8775-6df4bb3bcd2a\" (UID: \"37aa955b-ef9d-4016-8775-6df4bb3bcd2a\") " Nov 02 20:19:22.003408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:22.003392 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-kube-api-access-dbsf2\" (UniqueName: \"kubernetes.io/projected/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-kube-api-access-dbsf2\") pod \"37aa955b-ef9d-4016-8775-6df4bb3bcd2a\" (UID: \"37aa955b-ef9d-4016-8775-6df4bb3bcd2a\") " Nov 02 20:19:22.003654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:22.003629 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-plugins-config-volume" pod "37aa955b-ef9d-4016-8775-6df4bb3bcd2a" (UID: "37aa955b-ef9d-4016-8775-6df4bb3bcd2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:22.004025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:22.004007 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:22.005347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:22.005324 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-kube-api-access-dbsf2" pod "37aa955b-ef9d-4016-8775-6df4bb3bcd2a" (UID: "37aa955b-ef9d-4016-8775-6df4bb3bcd2a"). InnerVolumeSpecName "kube-api-access-dbsf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:22.106030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:22.105998 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbsf2\" (UniqueName: \"kubernetes.io/projected/37aa955b-ef9d-4016-8775-6df4bb3bcd2a-kube-api-access-dbsf2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:22.407860 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:22.407769 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks"] Nov 02 20:19:22.412011 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:22.411983 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7f75722a-547d6b69df-r4nks"] Nov 02 20:19:23.000553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:23.000517 5399 generic.go:356] "Generic (PLEG): container finished" podID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerID="25bcc211fe6c441dd174c2e2c7cd8d989b48d4b5fcf7da6d908284dd60adc4eb" exitCode=0 Nov 02 20:19:23.001081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:23.000596 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" event={"ID":"56fec1f1-6034-4946-92b3-45d90dc143d8","Type":"ContainerDied","Data":"25bcc211fe6c441dd174c2e2c7cd8d989b48d4b5fcf7da6d908284dd60adc4eb"} Nov 02 20:19:23.001219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:23.001199 5399 scope.go:122] "RemoveContainer" containerID="25bcc211fe6c441dd174c2e2c7cd8d989b48d4b5fcf7da6d908284dd60adc4eb" Nov 02 20:19:23.002916 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:23.002888 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:23.003018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:23.002954 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:23.003018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:23.002987 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:23.045215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:23.045184 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5"] Nov 02 20:19:23.092126 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:23.092100 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba11d4ae-6qwp5"] Nov 02 20:19:23.519716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:23.519677 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251aa06c-6ddf-4d9c-afd2-134515a1becb" path="/var/lib/kubelet/pods/251aa06c-6ddf-4d9c-afd2-134515a1becb/volumes" Nov 02 20:19:23.520606 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:23.520566 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37aa955b-ef9d-4016-8775-6df4bb3bcd2a" path="/var/lib/kubelet/pods/37aa955b-ef9d-4016-8775-6df4bb3bcd2a/volumes" Nov 02 20:19:24.011235 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:24.011205 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" event={"ID":"56fec1f1-6034-4946-92b3-45d90dc143d8","Type":"ContainerStarted","Data":"ec1ad8ed8589de5b40fcb74e57ecd164ebcd9aa152871cd780ae40ff28718916"} Nov 02 20:19:24.759065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:24.759022 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:19:24.759329 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:24.759309 5399 patch_prober.go:29] interesting pod/e2e-ait-13c1d955-openshift-default-689646446c-q87fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 20:19:24.759397 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:24.759362 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 20:19:25.758888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:25.758854 5399 patch_prober.go:29] interesting pod/e2e-ait-13c1d955-openshift-default-689646446c-q87fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 20:19:25.759314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:25.758918 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 20:19:26.759349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:26.759309 5399 patch_prober.go:29] interesting pod/e2e-ait-13c1d955-openshift-default-689646446c-q87fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 20:19:26.759842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:26.759371 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 20:19:27.759220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:27.759182 5399 patch_prober.go:29] interesting pod/e2e-ait-13c1d955-openshift-default-689646446c-q87fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 20:19:27.759428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:27.759237 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 20:19:28.759501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:28.759459 5399 patch_prober.go:29] interesting pod/e2e-ait-13c1d955-openshift-default-689646446c-q87fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 02 20:19:28.759925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:28.759522 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 02 20:19:29.764011 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:29.763979 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:19:29.764494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:29.764469 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:19:29.765243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:29.765219 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:19:39.044621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:39.044585 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46"] Nov 02 20:19:39.091266 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:39.091228 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6fc5f80-q7z46"] Nov 02 20:19:39.509172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:39.509137 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762b1976-bfc9-4e3e-a44e-a5c8c65f7bae" path="/var/lib/kubelet/pods/762b1976-bfc9-4e3e-a44e-a5c8c65f7bae/volumes" Nov 02 20:19:46.118897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:46.118859 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr"] Nov 02 20:19:46.122820 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:46.122793 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189c82ec-t7mrr"] Nov 02 20:19:47.532503 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:47.532358 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c06fb08-a82b-4c43-924a-253cc7fec830" path="/var/lib/kubelet/pods/6c06fb08-a82b-4c43-924a-253cc7fec830/volumes" Nov 02 20:19:48.245923 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:19:48.245883 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:19:49.208109 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:49.208076 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c"] Nov 02 20:19:49.208517 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:49.208321 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" containerID="cri-o://09214e2d9acfea55be9fdfaf031a85b57db48b51307fcfe75f1c6b94ccd17726" gracePeriod=30 Nov 02 20:19:54.212162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.212122 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:19:54.347592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.347548 5399 generic.go:356] "Generic (PLEG): container finished" podID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerID="09214e2d9acfea55be9fdfaf031a85b57db48b51307fcfe75f1c6b94ccd17726" exitCode=0 Nov 02 20:19:54.347758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.347618 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" event={"ID":"88b51926-7345-4f60-b2b7-0a82fbcca4c5","Type":"ContainerDied","Data":"09214e2d9acfea55be9fdfaf031a85b57db48b51307fcfe75f1c6b94ccd17726"} Nov 02 20:19:54.518684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.518647 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:19:54.622819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.622786 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-envoy\") pod \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " Nov 02 20:19:54.622995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.622872 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-kube-api-access-ssxml\" (UniqueName: \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-kube-api-access-ssxml\") pod \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " Nov 02 20:19:54.623065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623005 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-data\") pod \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " Nov 02 20:19:54.623124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623078 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-token\" (UniqueName: \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-token\") pod \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " Nov 02 20:19:54.623124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623116 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-socket\") pod \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " Nov 02 20:19:54.623235 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623168 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-certs\") pod \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " Nov 02 20:19:54.623235 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623208 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istiod-ca-cert\") pod \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " Nov 02 20:19:54.623370 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623327 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-data" pod "88b51926-7345-4f60-b2b7-0a82fbcca4c5" (UID: "88b51926-7345-4f60-b2b7-0a82fbcca4c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:54.623491 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623460 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-podinfo\") pod \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " Nov 02 20:19:54.623659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623641 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-credential-socket\") pod \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\" (UID: \"88b51926-7345-4f60-b2b7-0a82fbcca4c5\") " Nov 02 20:19:54.623840 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623554 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-certs" pod "88b51926-7345-4f60-b2b7-0a82fbcca4c5" (UID: "88b51926-7345-4f60-b2b7-0a82fbcca4c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:54.623913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623655 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istiod-ca-cert" pod "88b51926-7345-4f60-b2b7-0a82fbcca4c5" (UID: "88b51926-7345-4f60-b2b7-0a82fbcca4c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:54.623913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623775 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-socket" pod "88b51926-7345-4f60-b2b7-0a82fbcca4c5" (UID: "88b51926-7345-4f60-b2b7-0a82fbcca4c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:54.624039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.623911 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-credential-socket" pod "88b51926-7345-4f60-b2b7-0a82fbcca4c5" (UID: "88b51926-7345-4f60-b2b7-0a82fbcca4c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:54.624917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.624885 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:54.625056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.624921 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:54.625056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.624936 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:54.625056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.624950 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:54.625056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.624965 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:54.625761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.625541 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-envoy" pod "88b51926-7345-4f60-b2b7-0a82fbcca4c5" (UID: "88b51926-7345-4f60-b2b7-0a82fbcca4c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:54.625761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.625541 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-kube-api-access-ssxml" pod "88b51926-7345-4f60-b2b7-0a82fbcca4c5" (UID: "88b51926-7345-4f60-b2b7-0a82fbcca4c5"). InnerVolumeSpecName "kube-api-access-ssxml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:54.625761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.625729 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-podinfo" pod "88b51926-7345-4f60-b2b7-0a82fbcca4c5" (UID: "88b51926-7345-4f60-b2b7-0a82fbcca4c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:54.625954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.625939 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-token" pod "88b51926-7345-4f60-b2b7-0a82fbcca4c5" (UID: "88b51926-7345-4f60-b2b7-0a82fbcca4c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:54.727250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.727210 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:54.727250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.727250 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:54.727530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.727265 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b51926-7345-4f60-b2b7-0a82fbcca4c5-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:54.727530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:54.727317 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssxml\" (UniqueName: \"kubernetes.io/projected/88b51926-7345-4f60-b2b7-0a82fbcca4c5-kube-api-access-ssxml\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:55.359599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:55.359564 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" Nov 02 20:19:55.359599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:55.359580 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" event={"ID":"88b51926-7345-4f60-b2b7-0a82fbcca4c5","Type":"ContainerDied","Data":"0d90e6b1cc1e665385dbaa14f186a025b6607b1a685970c7a491293c3b6a768e"} Nov 02 20:19:55.360117 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:55.359655 5399 scope.go:122] "RemoveContainer" containerID="09214e2d9acfea55be9fdfaf031a85b57db48b51307fcfe75f1c6b94ccd17726" Nov 02 20:19:55.431776 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:55.431736 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:19:55.431977 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:55.431821 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:19:55.566392 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:55.566356 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c"] Nov 02 20:19:55.603423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:55.603379 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-kjx8c"] Nov 02 20:19:57.521462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:57.521421 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" path="/var/lib/kubelet/pods/88b51926-7345-4f60-b2b7-0a82fbcca4c5/volumes" Nov 02 20:19:59.056676 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:59.056639 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl"] Nov 02 20:19:59.102423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:59.102388 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fde7750e-lqxsl"] Nov 02 20:19:59.512058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:19:59.511972 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21079db8-15e7-4e85-9582-c4863ccee643" path="/var/lib/kubelet/pods/21079db8-15e7-4e85-9582-c4863ccee643/volumes" Nov 02 20:20:04.397041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:04.397002 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc"] Nov 02 20:20:05.019432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:05.018541 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": read tcp 10.133.0.2:49222->10.133.0.114:15021: read: connection reset by peer" start-of-body= Nov 02 20:20:05.019432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:05.018623 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": read tcp 10.133.0.2:49222->10.133.0.114:15021: read: connection reset by peer" Nov 02 20:20:05.447930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:05.447895 5399 generic.go:356] "Generic (PLEG): container finished" podID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerID="a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a" exitCode=0 Nov 02 20:20:05.448393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:05.447946 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" event={"ID":"5fd261a1-a473-4c79-a8d8-71d19923c944","Type":"ContainerDied","Data":"a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a"} Nov 02 20:20:05.448393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:05.448384 5399 scope.go:122] "RemoveContainer" containerID="a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a" Nov 02 20:20:05.449956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:05.449924 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:20:05.450026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:05.449998 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:20:05.450062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:05.450035 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:20:06.482301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.481375 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" event={"ID":"5fd261a1-a473-4c79-a8d8-71d19923c944","Type":"ContainerStarted","Data":"9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca"} Nov 02 20:20:06.792346 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.789379 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:20:06.792346 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.789506 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 20:20:06.792346 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.789561 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 20:20:06.795613 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.795579 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht"] Nov 02 20:20:06.796969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.796936 5399 cpu_manager.go:414] "containerMap: removing container" podUID="37aa955b-ef9d-4016-8775-6df4bb3bcd2a" containerName="payload-processing" Nov 02 20:20:06.797147 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.797130 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37aa955b-ef9d-4016-8775-6df4bb3bcd2a" containerName="payload-processing" Nov 02 20:20:06.797288 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.797256 5399 cpu_manager.go:414] "containerMap: removing container" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" Nov 02 20:20:06.797390 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.797375 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" Nov 02 20:20:06.797697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.797679 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="37aa955b-ef9d-4016-8775-6df4bb3bcd2a" containerName="payload-processing" Nov 02 20:20:06.797811 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.797801 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b51926-7345-4f60-b2b7-0a82fbcca4c5" containerName="istio-proxy" Nov 02 20:20:06.814477 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.812345 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:06.843560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.843513 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht"] Nov 02 20:20:06.867573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.867526 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-602be312-serving-cert\"" Nov 02 20:20:06.975671 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.975635 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-metrics-certs\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:06.975839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.975760 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc9560d7-7263-49b9-99e4-012124e9a461-openshift-service-ca\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:06.975941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.975921 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqk7\" (UniqueName: \"kubernetes.io/projected/dc9560d7-7263-49b9-99e4-012124e9a461-kube-api-access-7bqk7\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:06.976027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:06.976014 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:07.077654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:07.077614 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc9560d7-7263-49b9-99e4-012124e9a461-openshift-service-ca\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:07.077860 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:07.077780 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqk7\" (UniqueName: \"kubernetes.io/projected/dc9560d7-7263-49b9-99e4-012124e9a461-kube-api-access-7bqk7\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:07.077935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:07.077894 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:07.078017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:07.078002 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-metrics-certs\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:07.078693 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:07.078666 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-602be312-serving-cert: secret "maas-api-e2e-mt-602be312-serving-cert" not found Nov 02 20:20:07.078932 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:07.078917 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls podName:dc9560d7-7263-49b9-99e4-012124e9a461 nodeName:}" failed. No retries permitted until 2026-11-02 20:20:07.578896059 +0000 UTC m=+1650.189548134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls") pod "maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" (UID: "dc9560d7-7263-49b9-99e4-012124e9a461") : secret "maas-api-e2e-mt-602be312-serving-cert" not found Nov 02 20:20:07.079325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:07.079304 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc9560d7-7263-49b9-99e4-012124e9a461-openshift-service-ca\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:07.084328 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:07.080962 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-metrics-certs\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:07.149310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:07.148149 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqk7\" (UniqueName: \"kubernetes.io/projected/dc9560d7-7263-49b9-99e4-012124e9a461-kube-api-access-7bqk7\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:07.594149 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:07.594114 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:07.594690 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:07.594357 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-602be312-serving-cert: secret "maas-api-e2e-mt-602be312-serving-cert" not found Nov 02 20:20:07.594690 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:07.594433 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls podName:dc9560d7-7263-49b9-99e4-012124e9a461 nodeName:}" failed. No retries permitted until 2026-11-02 20:20:08.59441511 +0000 UTC m=+1651.205067186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls") pod "maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" (UID: "dc9560d7-7263-49b9-99e4-012124e9a461") : secret "maas-api-e2e-mt-602be312-serving-cert" not found Nov 02 20:20:07.789895 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:07.789863 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 20:20:07.790051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:07.789913 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 20:20:08.609163 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:08.609125 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:08.611705 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:08.611675 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls\") pod \"maas-api-e2e-mt-602be312-5597fb8dbc-nmjht\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:08.651586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:08.651555 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:08.790682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:08.790176 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 20:20:08.790682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:08.790251 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 20:20:09.052719 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:20:09.052670 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9560d7_7263_49b9_99e4_012124e9a461.slice/crio-81ae9d65a3909690813f9fbeddf523e9f8d419e9c259d90eeccc5f96d380aa7e WatchSource:0}: Error finding container 81ae9d65a3909690813f9fbeddf523e9f8d419e9c259d90eeccc5f96d380aa7e: Status 404 returned error can't find the container with id 81ae9d65a3909690813f9fbeddf523e9f8d419e9c259d90eeccc5f96d380aa7e Nov 02 20:20:09.053621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.053592 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht"] Nov 02 20:20:09.054873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.054850 5399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:20:09.397965 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.397912 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" podUID="8b1637a4-b052-43a4-be4d-f4e4f0d2ae80" containerName="payload-processing" containerID="cri-o://2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9" gracePeriod=25 Nov 02 20:20:09.516473 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.516430 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" event={"ID":"dc9560d7-7263-49b9-99e4-012124e9a461","Type":"ContainerStarted","Data":"81ae9d65a3909690813f9fbeddf523e9f8d419e9c259d90eeccc5f96d380aa7e"} Nov 02 20:20:09.765119 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.765089 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:20:09.789611 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.789586 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 20:20:09.789772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.789635 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 20:20:09.861837 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.861797 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-kube-api-access-k49gj\" (UniqueName: \"kubernetes.io/projected/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-kube-api-access-k49gj\") pod \"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80\" (UID: \"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80\") " Nov 02 20:20:09.862684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.862656 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-plugins-config-volume\") pod \"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80\" (UID: \"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80\") " Nov 02 20:20:09.864153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.864127 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-plugins-config-volume" pod "8b1637a4-b052-43a4-be4d-f4e4f0d2ae80" (UID: "8b1637a4-b052-43a4-be4d-f4e4f0d2ae80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:20:09.886392 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.886338 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-kube-api-access-k49gj" pod "8b1637a4-b052-43a4-be4d-f4e4f0d2ae80" (UID: "8b1637a4-b052-43a4-be4d-f4e4f0d2ae80"). InnerVolumeSpecName "kube-api-access-k49gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:09.966616 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.966493 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:09.966616 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:09.966531 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k49gj\" (UniqueName: \"kubernetes.io/projected/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80-kube-api-access-k49gj\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:10.549301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.548898 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" event={"ID":"dc9560d7-7263-49b9-99e4-012124e9a461","Type":"ContainerStarted","Data":"aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70"} Nov 02 20:20:10.551300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.550212 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:10.554457 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.554427 5399 generic.go:356] "Generic (PLEG): container finished" podID="8b1637a4-b052-43a4-be4d-f4e4f0d2ae80" containerID="2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9" exitCode=0 Nov 02 20:20:10.554579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.554515 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" Nov 02 20:20:10.554579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.554520 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" event={"ID":"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80","Type":"ContainerDied","Data":"2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9"} Nov 02 20:20:10.554703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.554577 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc" event={"ID":"8b1637a4-b052-43a4-be4d-f4e4f0d2ae80","Type":"ContainerDied","Data":"161da23823bed43dc0b14437693b08e86586c450d7dad6bd3f210d8c6471765a"} Nov 02 20:20:10.554703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.554610 5399 scope.go:122] "RemoveContainer" containerID="2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9" Nov 02 20:20:10.565766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.565743 5399 scope.go:122] "RemoveContainer" containerID="2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9" Nov 02 20:20:10.566065 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:10.566046 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9\": container with ID starting with 2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9 not found: ID does not exist" containerID="2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9" Nov 02 20:20:10.566116 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.566075 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9"} err="failed to get container status \"2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9\": rpc error: code = NotFound desc = could not find container \"2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9\": container with ID starting with 2e61b67e22adf05ee1767ec8bb54e887ddb4d4fec44be0dde4aaad079287cbb9 not found: ID does not exist" Nov 02 20:20:10.659379 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.659324 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" podStartSLOduration=3.6940424099999998 podStartE2EDuration="4.659268946s" totalImagesPullingTime="965.226536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:10.656619034 +0000 UTC m=+1653.267271125" watchObservedRunningTime="2026-11-02 20:20:10.659268946 +0000 UTC m=+1653.269921036" Nov 02 20:20:10.762656 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.762622 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc"] Nov 02 20:20:10.767133 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.767099 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55e81986-7448f4bb7c-pmcfc"] Nov 02 20:20:10.789584 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.789547 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 20:20:10.789771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:10.789605 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 20:20:11.057378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.057327 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h"] Nov 02 20:20:11.060835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.060809 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e003b6ec-ggr7h"] Nov 02 20:20:11.431360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.430588 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42"] Nov 02 20:20:11.431360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.430840 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" podUID="b3507784-bbfa-439c-bc46-e2e62076cc3a" containerName="maas-api" containerID="cri-o://96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04" gracePeriod=30 Nov 02 20:20:11.607012 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.604856 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ce1fd1-4f23-4b6b-9242-91ff525fae58" path="/var/lib/kubelet/pods/76ce1fd1-4f23-4b6b-9242-91ff525fae58/volumes" Nov 02 20:20:11.607012 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.605511 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1637a4-b052-43a4-be4d-f4e4f0d2ae80" path="/var/lib/kubelet/pods/8b1637a4-b052-43a4-be4d-f4e4f0d2ae80/volumes" Nov 02 20:20:11.790157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.790117 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 02 20:20:11.790700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.790172 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 02 20:20:11.844119 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.844091 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:20:11.894155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.894077 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3507784-bbfa-439c-bc46-e2e62076cc3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3507784-bbfa-439c-bc46-e2e62076cc3a-openshift-service-ca\") pod \"b3507784-bbfa-439c-bc46-e2e62076cc3a\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " Nov 02 20:20:11.894155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.894146 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3507784-bbfa-439c-bc46-e2e62076cc3a-kube-api-access-d48jh\" (UniqueName: \"kubernetes.io/projected/b3507784-bbfa-439c-bc46-e2e62076cc3a-kube-api-access-d48jh\") pod \"b3507784-bbfa-439c-bc46-e2e62076cc3a\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " Nov 02 20:20:11.896099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.895061 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3507784-bbfa-439c-bc46-e2e62076cc3a-openshift-service-ca" pod "b3507784-bbfa-439c-bc46-e2e62076cc3a" (UID: "b3507784-bbfa-439c-bc46-e2e62076cc3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:20:11.915297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.906446 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-maas-api-tls\") pod \"b3507784-bbfa-439c-bc46-e2e62076cc3a\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " Nov 02 20:20:11.915297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.906573 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-metrics-certs\") pod \"b3507784-bbfa-439c-bc46-e2e62076cc3a\" (UID: \"b3507784-bbfa-439c-bc46-e2e62076cc3a\") " Nov 02 20:20:11.915297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.907420 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3507784-bbfa-439c-bc46-e2e62076cc3a-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:11.915297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.911882 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-maas-api-tls" pod "b3507784-bbfa-439c-bc46-e2e62076cc3a" (UID: "b3507784-bbfa-439c-bc46-e2e62076cc3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:20:11.915297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.912192 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-metrics-certs" pod "b3507784-bbfa-439c-bc46-e2e62076cc3a" (UID: "b3507784-bbfa-439c-bc46-e2e62076cc3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:20:11.915297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:11.913871 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3507784-bbfa-439c-bc46-e2e62076cc3a-kube-api-access-d48jh" pod "b3507784-bbfa-439c-bc46-e2e62076cc3a" (UID: "b3507784-bbfa-439c-bc46-e2e62076cc3a"). InnerVolumeSpecName "kube-api-access-d48jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:12.009409 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.009314 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:12.009409 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.009353 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3507784-bbfa-439c-bc46-e2e62076cc3a-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:12.009409 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.009367 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d48jh\" (UniqueName: \"kubernetes.io/projected/b3507784-bbfa-439c-bc46-e2e62076cc3a-kube-api-access-d48jh\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:12.614221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.614188 5399 generic.go:356] "Generic (PLEG): container finished" podID="b3507784-bbfa-439c-bc46-e2e62076cc3a" containerID="96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04" exitCode=0 Nov 02 20:20:12.614425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.614235 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" Nov 02 20:20:12.614425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.614301 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" event={"ID":"b3507784-bbfa-439c-bc46-e2e62076cc3a","Type":"ContainerDied","Data":"96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04"} Nov 02 20:20:12.614425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.614354 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42" event={"ID":"b3507784-bbfa-439c-bc46-e2e62076cc3a","Type":"ContainerDied","Data":"73d8b8a6f747a1022de21f3347c78d256b50bdd89d09929e27b8561cc21f7b0a"} Nov 02 20:20:12.614425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.614375 5399 scope.go:122] "RemoveContainer" containerID="96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04" Nov 02 20:20:12.628027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.628003 5399 scope.go:122] "RemoveContainer" containerID="96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04" Nov 02 20:20:12.628450 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:12.628418 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04\": container with ID starting with 96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04 not found: ID does not exist" containerID="96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04" Nov 02 20:20:12.628583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.628461 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04"} err="failed to get container status \"96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04\": rpc error: code = NotFound desc = could not find container \"96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04\": container with ID starting with 96f4228709f01f9d14dd267b191764a5515e6c50b0f23a5450ec69a9e5b44d04 not found: ID does not exist" Nov 02 20:20:12.773467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.773428 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42"] Nov 02 20:20:12.794006 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.793974 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:20:12.794434 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.794419 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:20:12.795171 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.795145 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:20:12.816340 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:12.816314 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7f75722a-7dcdfdc7cf-ncz42"] Nov 02 20:20:13.516408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:13.516364 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3507784-bbfa-439c-bc46-e2e62076cc3a" path="/var/lib/kubelet/pods/b3507784-bbfa-439c-bc46-e2e62076cc3a/volumes" Nov 02 20:20:17.619826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:17.619795 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:20:18.242189 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:18.242152 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:20:27.048061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.048027 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv"] Nov 02 20:20:27.048600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.048248 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" podUID="d6dd6445-0aab-421f-ad16-0378b915f725" containerName="maas-api" containerID="cri-o://ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543" gracePeriod=30 Nov 02 20:20:27.353675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.353643 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:20:27.466601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.466545 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6dd6445-0aab-421f-ad16-0378b915f725-kube-api-access-lqpzc\" (UniqueName: \"kubernetes.io/projected/d6dd6445-0aab-421f-ad16-0378b915f725-kube-api-access-lqpzc\") pod \"d6dd6445-0aab-421f-ad16-0378b915f725\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " Nov 02 20:20:27.466846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.466657 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6dd6445-0aab-421f-ad16-0378b915f725-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dd6445-0aab-421f-ad16-0378b915f725-openshift-service-ca\") pod \"d6dd6445-0aab-421f-ad16-0378b915f725\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " Nov 02 20:20:27.466846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.466704 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-metrics-certs\") pod \"d6dd6445-0aab-421f-ad16-0378b915f725\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " Nov 02 20:20:27.467039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.466971 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-maas-api-tls\") pod \"d6dd6445-0aab-421f-ad16-0378b915f725\" (UID: \"d6dd6445-0aab-421f-ad16-0378b915f725\") " Nov 02 20:20:27.468658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.468625 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6dd6445-0aab-421f-ad16-0378b915f725-openshift-service-ca" pod "d6dd6445-0aab-421f-ad16-0378b915f725" (UID: "d6dd6445-0aab-421f-ad16-0378b915f725"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:20:27.470613 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.470581 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-metrics-certs" pod "d6dd6445-0aab-421f-ad16-0378b915f725" (UID: "d6dd6445-0aab-421f-ad16-0378b915f725"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:20:27.471198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.471169 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-maas-api-tls" pod "d6dd6445-0aab-421f-ad16-0378b915f725" (UID: "d6dd6445-0aab-421f-ad16-0378b915f725"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:20:27.471485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.471460 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6dd6445-0aab-421f-ad16-0378b915f725-kube-api-access-lqpzc" pod "d6dd6445-0aab-421f-ad16-0378b915f725" (UID: "d6dd6445-0aab-421f-ad16-0378b915f725"). InnerVolumeSpecName "kube-api-access-lqpzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:27.572801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.572704 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:27.572801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.572755 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqpzc\" (UniqueName: \"kubernetes.io/projected/d6dd6445-0aab-421f-ad16-0378b915f725-kube-api-access-lqpzc\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:27.572801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.572770 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dd6445-0aab-421f-ad16-0378b915f725-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:27.572801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.572786 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd6445-0aab-421f-ad16-0378b915f725-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:27.738156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.738119 5399 generic.go:356] "Generic (PLEG): container finished" podID="d6dd6445-0aab-421f-ad16-0378b915f725" containerID="ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543" exitCode=0 Nov 02 20:20:27.738371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.738210 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" Nov 02 20:20:27.738461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.738213 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" event={"ID":"d6dd6445-0aab-421f-ad16-0378b915f725","Type":"ContainerDied","Data":"ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543"} Nov 02 20:20:27.738545 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.738528 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv" event={"ID":"d6dd6445-0aab-421f-ad16-0378b915f725","Type":"ContainerDied","Data":"65afe7cf6545ec0da8bb4d77aaf572dc4812ff5ac77f3ccdce87d4aaf0f41d20"} Nov 02 20:20:27.738600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.738563 5399 scope.go:122] "RemoveContainer" containerID="ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543" Nov 02 20:20:27.761473 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.757891 5399 scope.go:122] "RemoveContainer" containerID="ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543" Nov 02 20:20:27.761473 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:27.758251 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543\": container with ID starting with ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543 not found: ID does not exist" containerID="ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543" Nov 02 20:20:27.761473 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.758319 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543"} err="failed to get container status \"ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543\": rpc error: code = NotFound desc = could not find container \"ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543\": container with ID starting with ba6b72928773da0b28f055a2b8054a741bee8be1525f01897e21671cf2b52543 not found: ID does not exist" Nov 02 20:20:27.895219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.895127 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv"] Nov 02 20:20:27.900545 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:27.900511 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13c1d955-6c6c6b75d8-l4pjv"] Nov 02 20:20:29.509878 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:29.509848 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6dd6445-0aab-421f-ad16-0378b915f725" path="/var/lib/kubelet/pods/d6dd6445-0aab-421f-ad16-0378b915f725/volumes" Nov 02 20:20:30.616474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:30.616436 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv"] Nov 02 20:20:30.616867 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:30.616654 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" containerID="cri-o://ec1ad8ed8589de5b40fcb74e57ecd164ebcd9aa152871cd780ae40ff28718916" gracePeriod=30 Nov 02 20:20:35.440722 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.440688 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w"] Nov 02 20:20:35.441159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.440918 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" containerID="cri-o://cbc248d79450376a5ea402f5444c052d228f6148f1951431954125a176ae2002" gracePeriod=30 Nov 02 20:20:35.822519 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.822487 5399 generic.go:356] "Generic (PLEG): container finished" podID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerID="ec1ad8ed8589de5b40fcb74e57ecd164ebcd9aa152871cd780ae40ff28718916" exitCode=0 Nov 02 20:20:35.822694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.822558 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" event={"ID":"56fec1f1-6034-4946-92b3-45d90dc143d8","Type":"ContainerDied","Data":"ec1ad8ed8589de5b40fcb74e57ecd164ebcd9aa152871cd780ae40ff28718916"} Nov 02 20:20:35.822694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.822621 5399 scope.go:122] "RemoveContainer" containerID="25bcc211fe6c441dd174c2e2c7cd8d989b48d4b5fcf7da6d908284dd60adc4eb" Nov 02 20:20:35.911825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.911800 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:20:35.957442 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957411 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-token\" (UniqueName: \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-token\") pod \"56fec1f1-6034-4946-92b3-45d90dc143d8\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " Nov 02 20:20:35.957596 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957512 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-credential-socket\") pod \"56fec1f1-6034-4946-92b3-45d90dc143d8\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " Nov 02 20:20:35.957596 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957536 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-podinfo\") pod \"56fec1f1-6034-4946-92b3-45d90dc143d8\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " Nov 02 20:20:35.957596 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957589 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-socket\") pod \"56fec1f1-6034-4946-92b3-45d90dc143d8\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " Nov 02 20:20:35.957731 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957624 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-kube-api-access-6xsss\" (UniqueName: \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-kube-api-access-6xsss\") pod \"56fec1f1-6034-4946-92b3-45d90dc143d8\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " Nov 02 20:20:35.957731 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957692 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fec1f1-6034-4946-92b3-45d90dc143d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fec1f1-6034-4946-92b3-45d90dc143d8-istiod-ca-cert\") pod \"56fec1f1-6034-4946-92b3-45d90dc143d8\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " Nov 02 20:20:35.957845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957765 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-certs\") pod \"56fec1f1-6034-4946-92b3-45d90dc143d8\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " Nov 02 20:20:35.957845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957808 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-envoy\") pod \"56fec1f1-6034-4946-92b3-45d90dc143d8\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " Nov 02 20:20:35.957845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957824 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-credential-socket" pod "56fec1f1-6034-4946-92b3-45d90dc143d8" (UID: "56fec1f1-6034-4946-92b3-45d90dc143d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:35.957994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957842 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-data\") pod \"56fec1f1-6034-4946-92b3-45d90dc143d8\" (UID: \"56fec1f1-6034-4946-92b3-45d90dc143d8\") " Nov 02 20:20:35.957994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.957918 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-socket" pod "56fec1f1-6034-4946-92b3-45d90dc143d8" (UID: "56fec1f1-6034-4946-92b3-45d90dc143d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:35.958088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.958045 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-data" pod "56fec1f1-6034-4946-92b3-45d90dc143d8" (UID: "56fec1f1-6034-4946-92b3-45d90dc143d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:35.958134 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.958111 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fec1f1-6034-4946-92b3-45d90dc143d8-istiod-ca-cert" pod "56fec1f1-6034-4946-92b3-45d90dc143d8" (UID: "56fec1f1-6034-4946-92b3-45d90dc143d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:20:35.958181 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.958162 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-certs" pod "56fec1f1-6034-4946-92b3-45d90dc143d8" (UID: "56fec1f1-6034-4946-92b3-45d90dc143d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:35.958707 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.958685 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:35.958808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.958709 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:35.958808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.958723 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56fec1f1-6034-4946-92b3-45d90dc143d8-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:35.958808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.958735 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:35.958808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.958747 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:35.959818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.959794 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-kube-api-access-6xsss" pod "56fec1f1-6034-4946-92b3-45d90dc143d8" (UID: "56fec1f1-6034-4946-92b3-45d90dc143d8"). InnerVolumeSpecName "kube-api-access-6xsss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:35.959990 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.959972 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-envoy" pod "56fec1f1-6034-4946-92b3-45d90dc143d8" (UID: "56fec1f1-6034-4946-92b3-45d90dc143d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:35.960087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.960064 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-podinfo" pod "56fec1f1-6034-4946-92b3-45d90dc143d8" (UID: "56fec1f1-6034-4946-92b3-45d90dc143d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:20:35.960140 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:35.960091 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-token" pod "56fec1f1-6034-4946-92b3-45d90dc143d8" (UID: "56fec1f1-6034-4946-92b3-45d90dc143d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:36.060084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:36.060047 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:36.060084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:36.060083 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xsss\" (UniqueName: \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-kube-api-access-6xsss\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:36.060387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:36.060100 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:36.060387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:36.060112 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56fec1f1-6034-4946-92b3-45d90dc143d8-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:36.473848 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:36.473562 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7"] Nov 02 20:20:36.473848 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:36.473790 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" containerID="cri-o://4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98" gracePeriod=30 Nov 02 20:20:36.832492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:36.832458 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" Nov 02 20:20:36.832690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:36.832458 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv" event={"ID":"56fec1f1-6034-4946-92b3-45d90dc143d8","Type":"ContainerDied","Data":"481160cb027d790c1641a6f5037b7533ea80f1480fa4b10347c78d0f06cb4ba3"} Nov 02 20:20:36.832690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:36.832593 5399 scope.go:122] "RemoveContainer" containerID="ec1ad8ed8589de5b40fcb74e57ecd164ebcd9aa152871cd780ae40ff28718916" Nov 02 20:20:36.999846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:36.999814 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv"] Nov 02 20:20:37.046670 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:37.046634 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-13c1d955-openshift-default-689646446c-q87fv"] Nov 02 20:20:37.510995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:37.510966 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" path="/var/lib/kubelet/pods/56fec1f1-6034-4946-92b3-45d90dc143d8/volumes" Nov 02 20:20:37.923176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:37.923144 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj"] Nov 02 20:20:37.923397 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:37.923352 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" podUID="33b4b611-a84f-4da7-bae2-e26d177a6753" containerName="maas-api" containerID="cri-o://0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1" gracePeriod=30 Nov 02 20:20:37.998218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:37.998187 5399 scope.go:122] "RemoveContainer" containerID="5e07fc1b36f58272d530202d39bcfe3c10c21e8d9c8442beed8e606fce57df6b" Nov 02 20:20:38.037816 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.037796 5399 scope.go:122] "RemoveContainer" containerID="0f516dcbf02159c99b586d40db96caa99f3ff1128c0eb1ff3200459117d8cca0" Nov 02 20:20:38.047182 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.047158 5399 scope.go:122] "RemoveContainer" containerID="cb460719902d3f74817de8c7181fdc4281050218a3f585664dc7663833d1f418" Nov 02 20:20:38.057028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.057009 5399 scope.go:122] "RemoveContainer" containerID="f6f219ad423ba67dc72f933ee107f7a04833c1881dd9d528c5a7fca962f8ee23" Nov 02 20:20:38.258354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.258323 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:20:38.301695 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.301659 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b4b611-a84f-4da7-bae2-e26d177a6753-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b4b611-a84f-4da7-bae2-e26d177a6753-openshift-service-ca\") pod \"33b4b611-a84f-4da7-bae2-e26d177a6753\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " Nov 02 20:20:38.301886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.301776 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-maas-api-tls\") pod \"33b4b611-a84f-4da7-bae2-e26d177a6753\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " Nov 02 20:20:38.301886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.301825 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b4b611-a84f-4da7-bae2-e26d177a6753-kube-api-access-snmbd\" (UniqueName: \"kubernetes.io/projected/33b4b611-a84f-4da7-bae2-e26d177a6753-kube-api-access-snmbd\") pod \"33b4b611-a84f-4da7-bae2-e26d177a6753\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " Nov 02 20:20:38.302016 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.301883 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-metrics-certs\") pod \"33b4b611-a84f-4da7-bae2-e26d177a6753\" (UID: \"33b4b611-a84f-4da7-bae2-e26d177a6753\") " Nov 02 20:20:38.302140 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.302111 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b4b611-a84f-4da7-bae2-e26d177a6753-openshift-service-ca" pod "33b4b611-a84f-4da7-bae2-e26d177a6753" (UID: "33b4b611-a84f-4da7-bae2-e26d177a6753"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:20:38.302697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.302669 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b4b611-a84f-4da7-bae2-e26d177a6753-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:38.304187 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.304155 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-maas-api-tls" pod "33b4b611-a84f-4da7-bae2-e26d177a6753" (UID: "33b4b611-a84f-4da7-bae2-e26d177a6753"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:20:38.304269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.304243 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b4b611-a84f-4da7-bae2-e26d177a6753-kube-api-access-snmbd" pod "33b4b611-a84f-4da7-bae2-e26d177a6753" (UID: "33b4b611-a84f-4da7-bae2-e26d177a6753"). InnerVolumeSpecName "kube-api-access-snmbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:38.304586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.304564 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-metrics-certs" pod "33b4b611-a84f-4da7-bae2-e26d177a6753" (UID: "33b4b611-a84f-4da7-bae2-e26d177a6753"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:20:38.404163 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.404129 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:38.404163 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.404162 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snmbd\" (UniqueName: \"kubernetes.io/projected/33b4b611-a84f-4da7-bae2-e26d177a6753-kube-api-access-snmbd\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:38.404406 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.404175 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33b4b611-a84f-4da7-bae2-e26d177a6753-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:38.850738 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.850702 5399 generic.go:356] "Generic (PLEG): container finished" podID="33b4b611-a84f-4da7-bae2-e26d177a6753" containerID="0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1" exitCode=0 Nov 02 20:20:38.851174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.850748 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" Nov 02 20:20:38.851174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.850781 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" event={"ID":"33b4b611-a84f-4da7-bae2-e26d177a6753","Type":"ContainerDied","Data":"0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1"} Nov 02 20:20:38.851174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.850837 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj" event={"ID":"33b4b611-a84f-4da7-bae2-e26d177a6753","Type":"ContainerDied","Data":"f2ad804957ff655a072c9d3c9bc94fdf79302e5206fca2e56278f1479c2aa61c"} Nov 02 20:20:38.851174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.850865 5399 scope.go:122] "RemoveContainer" containerID="0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1" Nov 02 20:20:38.865542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.865508 5399 scope.go:122] "RemoveContainer" containerID="0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1" Nov 02 20:20:38.865863 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:38.865844 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1\": container with ID starting with 0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1 not found: ID does not exist" containerID="0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1" Nov 02 20:20:38.865914 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:38.865872 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1"} err="failed to get container status \"0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1\": rpc error: code = NotFound desc = could not find container \"0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1\": container with ID starting with 0da0778c21d3df7691af571a0fb057d58024567c9242f44ec0d5ee83edcdbba1 not found: ID does not exist" Nov 02 20:20:39.007750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:39.007714 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj"] Nov 02 20:20:39.056491 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:39.056450 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55e81986-57bdbbf6db-8njrj"] Nov 02 20:20:39.509048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:39.509017 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b4b611-a84f-4da7-bae2-e26d177a6753" path="/var/lib/kubelet/pods/33b4b611-a84f-4da7-bae2-e26d177a6753/volumes" Nov 02 20:20:40.874543 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:40.874504 5399 generic.go:356] "Generic (PLEG): container finished" podID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerID="cbc248d79450376a5ea402f5444c052d228f6148f1951431954125a176ae2002" exitCode=0 Nov 02 20:20:40.874932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:40.874550 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" event={"ID":"03d1fe52-533d-4ace-b8bd-defa734d3278","Type":"ContainerDied","Data":"cbc248d79450376a5ea402f5444c052d228f6148f1951431954125a176ae2002"} Nov 02 20:20:41.237033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.237008 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:20:41.381731 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.381688 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-certs\") pod \"03d1fe52-533d-4ace-b8bd-defa734d3278\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " Nov 02 20:20:41.381936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.381776 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-data\") pod \"03d1fe52-533d-4ace-b8bd-defa734d3278\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " Nov 02 20:20:41.381936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.381860 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-credential-socket\") pod \"03d1fe52-533d-4ace-b8bd-defa734d3278\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " Nov 02 20:20:41.381936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.381891 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-podinfo\") pod \"03d1fe52-533d-4ace-b8bd-defa734d3278\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " Nov 02 20:20:41.382099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.381940 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-envoy\") pod \"03d1fe52-533d-4ace-b8bd-defa734d3278\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " Nov 02 20:20:41.382099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.381997 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03d1fe52-533d-4ace-b8bd-defa734d3278-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d1fe52-533d-4ace-b8bd-defa734d3278-istiod-ca-cert\") pod \"03d1fe52-533d-4ace-b8bd-defa734d3278\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " Nov 02 20:20:41.382099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.382018 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-certs" pod "03d1fe52-533d-4ace-b8bd-defa734d3278" (UID: "03d1fe52-533d-4ace-b8bd-defa734d3278"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:41.382099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.382066 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-kube-api-access-h9k57\" (UniqueName: \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-kube-api-access-h9k57\") pod \"03d1fe52-533d-4ace-b8bd-defa734d3278\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " Nov 02 20:20:41.382330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.382089 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-data" pod "03d1fe52-533d-4ace-b8bd-defa734d3278" (UID: "03d1fe52-533d-4ace-b8bd-defa734d3278"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:41.382330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.382119 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-credential-socket" pod "03d1fe52-533d-4ace-b8bd-defa734d3278" (UID: "03d1fe52-533d-4ace-b8bd-defa734d3278"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:41.382330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.382243 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-socket\") pod \"03d1fe52-533d-4ace-b8bd-defa734d3278\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " Nov 02 20:20:41.382502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.382353 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-token\" (UniqueName: \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-token\") pod \"03d1fe52-533d-4ace-b8bd-defa734d3278\" (UID: \"03d1fe52-533d-4ace-b8bd-defa734d3278\") " Nov 02 20:20:41.382502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.382479 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-socket" pod "03d1fe52-533d-4ace-b8bd-defa734d3278" (UID: "03d1fe52-533d-4ace-b8bd-defa734d3278"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:41.382607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.382585 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d1fe52-533d-4ace-b8bd-defa734d3278-istiod-ca-cert" pod "03d1fe52-533d-4ace-b8bd-defa734d3278" (UID: "03d1fe52-533d-4ace-b8bd-defa734d3278"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:20:41.383307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.383242 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.383307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.383287 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.383307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.383306 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.383537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.383323 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d1fe52-533d-4ace-b8bd-defa734d3278-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.383537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.383339 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.384373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.384342 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-envoy" pod "03d1fe52-533d-4ace-b8bd-defa734d3278" (UID: "03d1fe52-533d-4ace-b8bd-defa734d3278"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:41.384448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.384374 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-podinfo" pod "03d1fe52-533d-4ace-b8bd-defa734d3278" (UID: "03d1fe52-533d-4ace-b8bd-defa734d3278"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:20:41.384511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.384496 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-kube-api-access-h9k57" pod "03d1fe52-533d-4ace-b8bd-defa734d3278" (UID: "03d1fe52-533d-4ace-b8bd-defa734d3278"). InnerVolumeSpecName "kube-api-access-h9k57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:41.384604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.384591 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-token" pod "03d1fe52-533d-4ace-b8bd-defa734d3278" (UID: "03d1fe52-533d-4ace-b8bd-defa734d3278"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:41.484997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.484919 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.484997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.484948 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.484997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.484958 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d1fe52-533d-4ace-b8bd-defa734d3278-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.484997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.484969 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9k57\" (UniqueName: \"kubernetes.io/projected/03d1fe52-533d-4ace-b8bd-defa734d3278-kube-api-access-h9k57\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.755505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.755449 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:20:41.884200 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.884167 5399 generic.go:356] "Generic (PLEG): container finished" podID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerID="4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98" exitCode=0 Nov 02 20:20:41.884680 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.884234 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" Nov 02 20:20:41.884680 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.884255 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" event={"ID":"1cf5b8ba-4340-4df2-9763-2c2f071862d5","Type":"ContainerDied","Data":"4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98"} Nov 02 20:20:41.884680 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.884336 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7" event={"ID":"1cf5b8ba-4340-4df2-9763-2c2f071862d5","Type":"ContainerDied","Data":"f82c78f85bc1ce06d7ea92c5d074d3cf8efe35f14d406188aa131ed98eb0225f"} Nov 02 20:20:41.884680 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.884369 5399 scope.go:122] "RemoveContainer" containerID="4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98" Nov 02 20:20:41.887144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.887126 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" Nov 02 20:20:41.887368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.887152 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w" event={"ID":"03d1fe52-533d-4ace-b8bd-defa734d3278","Type":"ContainerDied","Data":"bd7db870f665c0f26152f73e2afa13b7db23d2c5d0fc68e6cca9f7444dd724d4"} Nov 02 20:20:41.891145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.891122 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-podinfo\") pod \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " Nov 02 20:20:41.891351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.891329 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istiod-ca-cert\") pod \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " Nov 02 20:20:41.891468 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.891445 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-socket\") pod \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " Nov 02 20:20:41.891547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.891474 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-envoy\") pod \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " Nov 02 20:20:41.891547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.891534 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-data\") pod \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " Nov 02 20:20:41.891652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.891592 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-certs\") pod \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " Nov 02 20:20:41.891652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.891624 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-token\" (UniqueName: \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-token\") pod \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " Nov 02 20:20:41.891758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.891668 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-credential-socket\") pod \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " Nov 02 20:20:41.891758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.891710 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-kube-api-access-z47j2\" (UniqueName: \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-kube-api-access-z47j2\") pod \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\" (UID: \"1cf5b8ba-4340-4df2-9763-2c2f071862d5\") " Nov 02 20:20:41.892497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.892424 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-certs" pod "1cf5b8ba-4340-4df2-9763-2c2f071862d5" (UID: "1cf5b8ba-4340-4df2-9763-2c2f071862d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:41.892717 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.892508 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istiod-ca-cert" pod "1cf5b8ba-4340-4df2-9763-2c2f071862d5" (UID: "1cf5b8ba-4340-4df2-9763-2c2f071862d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:20:41.892893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.892857 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-credential-socket" pod "1cf5b8ba-4340-4df2-9763-2c2f071862d5" (UID: "1cf5b8ba-4340-4df2-9763-2c2f071862d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:41.893383 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.893351 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-socket" pod "1cf5b8ba-4340-4df2-9763-2c2f071862d5" (UID: "1cf5b8ba-4340-4df2-9763-2c2f071862d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:41.893658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.893639 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-data" pod "1cf5b8ba-4340-4df2-9763-2c2f071862d5" (UID: "1cf5b8ba-4340-4df2-9763-2c2f071862d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:41.894537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.894506 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-podinfo" pod "1cf5b8ba-4340-4df2-9763-2c2f071862d5" (UID: "1cf5b8ba-4340-4df2-9763-2c2f071862d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:20:41.895712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.895681 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-token" pod "1cf5b8ba-4340-4df2-9763-2c2f071862d5" (UID: "1cf5b8ba-4340-4df2-9763-2c2f071862d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:41.895796 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.895705 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-kube-api-access-z47j2" pod "1cf5b8ba-4340-4df2-9763-2c2f071862d5" (UID: "1cf5b8ba-4340-4df2-9763-2c2f071862d5"). InnerVolumeSpecName "kube-api-access-z47j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:41.895796 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.895760 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-envoy" pod "1cf5b8ba-4340-4df2-9763-2c2f071862d5" (UID: "1cf5b8ba-4340-4df2-9763-2c2f071862d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:41.899483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.899460 5399 scope.go:122] "RemoveContainer" containerID="4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98" Nov 02 20:20:41.899928 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:41.899896 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98\": container with ID starting with 4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98 not found: ID does not exist" containerID="4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98" Nov 02 20:20:41.900022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.899936 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98"} err="failed to get container status \"4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98\": rpc error: code = NotFound desc = could not find container \"4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98\": container with ID starting with 4db32733b79202bff5491fcf78cc3ee348a93e35ebe2a8aea5304539293c4f98 not found: ID does not exist" Nov 02 20:20:41.900022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.899963 5399 scope.go:122] "RemoveContainer" containerID="cbc248d79450376a5ea402f5444c052d228f6148f1951431954125a176ae2002" Nov 02 20:20:41.994160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.994127 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.994160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.994154 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.994160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.994166 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.994418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.994176 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.994418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.994185 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.994418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.994194 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf5b8ba-4340-4df2-9763-2c2f071862d5-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.994418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.994203 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z47j2\" (UniqueName: \"kubernetes.io/projected/1cf5b8ba-4340-4df2-9763-2c2f071862d5-kube-api-access-z47j2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.994418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.994213 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:41.994418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:41.994221 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf5b8ba-4340-4df2-9763-2c2f071862d5-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:42.044286 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.044200 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w"] Nov 02 20:20:42.050433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.050403 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-xsq6w"] Nov 02 20:20:42.059313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.059268 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj"] Nov 02 20:20:42.060301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060263 5399 cpu_manager.go:414] "containerMap: removing container" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" Nov 02 20:20:42.060350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060306 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" Nov 02 20:20:42.060350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060338 5399 cpu_manager.go:414] "containerMap: removing container" podUID="8b1637a4-b052-43a4-be4d-f4e4f0d2ae80" containerName="payload-processing" Nov 02 20:20:42.060350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060347 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1637a4-b052-43a4-be4d-f4e4f0d2ae80" containerName="payload-processing" Nov 02 20:20:42.060449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060359 5399 cpu_manager.go:414] "containerMap: removing container" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" Nov 02 20:20:42.060449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060367 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" Nov 02 20:20:42.060449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060385 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d6dd6445-0aab-421f-ad16-0378b915f725" containerName="maas-api" Nov 02 20:20:42.060449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060392 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6dd6445-0aab-421f-ad16-0378b915f725" containerName="maas-api" Nov 02 20:20:42.060449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060412 5399 cpu_manager.go:414] "containerMap: removing container" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" Nov 02 20:20:42.060449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060419 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" Nov 02 20:20:42.060449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060433 5399 cpu_manager.go:414] "containerMap: removing container" podUID="b3507784-bbfa-439c-bc46-e2e62076cc3a" containerName="maas-api" Nov 02 20:20:42.060449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060440 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3507784-bbfa-439c-bc46-e2e62076cc3a" containerName="maas-api" Nov 02 20:20:42.060663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060473 5399 cpu_manager.go:414] "containerMap: removing container" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" Nov 02 20:20:42.060663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060481 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" Nov 02 20:20:42.060663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060506 5399 cpu_manager.go:414] "containerMap: removing container" podUID="33b4b611-a84f-4da7-bae2-e26d177a6753" containerName="maas-api" Nov 02 20:20:42.060663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060514 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b4b611-a84f-4da7-bae2-e26d177a6753" containerName="maas-api" Nov 02 20:20:42.060774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060684 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1637a4-b052-43a4-be4d-f4e4f0d2ae80" containerName="payload-processing" Nov 02 20:20:42.060774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060699 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" Nov 02 20:20:42.060774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060711 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6dd6445-0aab-421f-ad16-0378b915f725" containerName="maas-api" Nov 02 20:20:42.060774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060725 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fec1f1-6034-4946-92b3-45d90dc143d8" containerName="istio-proxy" Nov 02 20:20:42.060774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060734 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3507784-bbfa-439c-bc46-e2e62076cc3a" containerName="maas-api" Nov 02 20:20:42.060774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060747 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" containerName="istio-proxy" Nov 02 20:20:42.060774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060758 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b4b611-a84f-4da7-bae2-e26d177a6753" containerName="maas-api" Nov 02 20:20:42.060774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.060771 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" containerName="istio-proxy" Nov 02 20:20:42.066714 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.066692 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.143598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.143329 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj"] Nov 02 20:20:42.161806 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.161758 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-5fc50d-openshift-default-dockercfg-wbdkt\"" Nov 02 20:20:42.198358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.198306 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6l5\" (UniqueName: \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-kube-api-access-bq6l5\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.198548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.198373 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-envoy\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.198614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.198585 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-token\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.198668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.198650 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-data\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.199215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.198745 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-certs\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.199215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.198777 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istiod-ca-cert\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.199215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.198858 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-socket\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.199215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.198899 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-credential-socket\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.199215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.199017 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-podinfo\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.303090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.303009 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-podinfo\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.303090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.303080 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6l5\" (UniqueName: \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-kube-api-access-bq6l5\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.303340 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.303127 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-envoy\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.303340 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.303227 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-token\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.303340 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.303285 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-data\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.303709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.303675 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-certs\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.303833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.303716 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istiod-ca-cert\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.303833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.303799 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-socket\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.303948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.303839 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-credential-socket\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.303948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.303893 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-data\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.304331 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.304292 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-socket\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.304507 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.304486 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istiod-ca-cert\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.304598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.304565 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-credential-socket\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.304757 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.304733 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-certs\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.305782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.305756 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-envoy\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.306065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.306043 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-podinfo\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.309070 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.309047 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7"] Nov 02 20:20:42.312685 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.312663 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-zw8s7"] Nov 02 20:20:42.355936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.355902 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-token\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.356084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.356004 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6l5\" (UniqueName: \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-kube-api-access-bq6l5\") pod \"e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.385565 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.385533 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:42.814466 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.814430 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj"] Nov 02 20:20:42.815402 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:20:42.815373 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe65bde2_a8b8_472b_bc2f_4a518e2702de.slice/crio-989ecfd89ac7bb5b4ad32fabd397a5d6e759e0203496e9070a019a09a2c338f4 WatchSource:0}: Error finding container 989ecfd89ac7bb5b4ad32fabd397a5d6e759e0203496e9070a019a09a2c338f4: Status 404 returned error can't find the container with id 989ecfd89ac7bb5b4ad32fabd397a5d6e759e0203496e9070a019a09a2c338f4 Nov 02 20:20:42.817790 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.817755 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:20:42.817872 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.817822 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:20:42.817872 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.817851 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:20:42.912708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:42.912614 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" event={"ID":"fe65bde2-a8b8-472b-bc2f-4a518e2702de","Type":"ContainerStarted","Data":"989ecfd89ac7bb5b4ad32fabd397a5d6e759e0203496e9070a019a09a2c338f4"} Nov 02 20:20:43.533081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:43.533042 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d1fe52-533d-4ace-b8bd-defa734d3278" path="/var/lib/kubelet/pods/03d1fe52-533d-4ace-b8bd-defa734d3278/volumes" Nov 02 20:20:43.533720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:43.533702 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf5b8ba-4340-4df2-9763-2c2f071862d5" path="/var/lib/kubelet/pods/1cf5b8ba-4340-4df2-9763-2c2f071862d5/volumes" Nov 02 20:20:43.918423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:43.918377 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" event={"ID":"fe65bde2-a8b8-472b-bc2f-4a518e2702de","Type":"ContainerStarted","Data":"703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8"} Nov 02 20:20:44.079405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:44.079348 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" podStartSLOduration=2.079334321 podStartE2EDuration="2.079334321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:44.07644821 +0000 UTC m=+1686.687100303" watchObservedRunningTime="2026-11-02 20:20:44.079334321 +0000 UTC m=+1686.689986412" Nov 02 20:20:44.386315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:44.386233 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:44.392518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:44.392489 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:44.928947 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:44.927431 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:44.928947 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:20:44.928766 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:20:48.241304 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:20:48.241255 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:21:05.348611 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:05.348574 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2"] Nov 02 20:21:05.349153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:05.348812 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" containerID="cri-o://a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913" gracePeriod=30 Nov 02 20:21:06.081560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:06.081523 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr"] Nov 02 20:21:06.081850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:06.081819 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" containerID="cri-o://c63567174de3f5f24853e05c5c3a982589e84a8612bf453cb0ee9bfba7a09071" gracePeriod=30 Nov 02 20:21:06.086027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:06.085996 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn"] Nov 02 20:21:06.086242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:06.086217 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" containerID="cri-o://cd997f825b4682dd93187ab4872aebf0127904aeba014c9eefa79089428fdfdf" gracePeriod=30 Nov 02 20:21:06.090542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:06.090509 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww"] Nov 02 20:21:06.090740 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:06.090714 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerName="istio-proxy" containerID="cri-o://92716e51a8ef9c94abe72a607c23245bcbae88c074edb177c9544283fe8e08e4" gracePeriod=30 Nov 02 20:21:09.722939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:09.722894 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:21:09.723404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:09.722986 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:21:10.697436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.697407 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:21:10.854118 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.854079 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-socket\") pod \"7c1d3714-9b66-41db-be48-ec8495162ff3\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " Nov 02 20:21:10.854704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.854670 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-data\") pod \"7c1d3714-9b66-41db-be48-ec8495162ff3\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " Nov 02 20:21:10.854827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.854724 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-token\" (UniqueName: \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-token\") pod \"7c1d3714-9b66-41db-be48-ec8495162ff3\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " Nov 02 20:21:10.854919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.854864 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-envoy\") pod \"7c1d3714-9b66-41db-be48-ec8495162ff3\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " Nov 02 20:21:10.854919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.854420 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-socket" pod "7c1d3714-9b66-41db-be48-ec8495162ff3" (UID: "7c1d3714-9b66-41db-be48-ec8495162ff3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:10.855010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.854887 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-data" pod "7c1d3714-9b66-41db-be48-ec8495162ff3" (UID: "7c1d3714-9b66-41db-be48-ec8495162ff3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:10.855010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.854919 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-certs\") pod \"7c1d3714-9b66-41db-be48-ec8495162ff3\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " Nov 02 20:21:10.855010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.854979 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-credential-socket\") pod \"7c1d3714-9b66-41db-be48-ec8495162ff3\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " Nov 02 20:21:10.855156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.855031 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-kube-api-access-d9r4f\" (UniqueName: \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-kube-api-access-d9r4f\") pod \"7c1d3714-9b66-41db-be48-ec8495162ff3\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " Nov 02 20:21:10.855156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.855144 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-certs" pod "7c1d3714-9b66-41db-be48-ec8495162ff3" (UID: "7c1d3714-9b66-41db-be48-ec8495162ff3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:10.855261 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.855221 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-podinfo\") pod \"7c1d3714-9b66-41db-be48-ec8495162ff3\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " Nov 02 20:21:10.855338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.855323 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1d3714-9b66-41db-be48-ec8495162ff3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1d3714-9b66-41db-be48-ec8495162ff3-istiod-ca-cert\") pod \"7c1d3714-9b66-41db-be48-ec8495162ff3\" (UID: \"7c1d3714-9b66-41db-be48-ec8495162ff3\") " Nov 02 20:21:10.856000 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.855976 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-credential-socket" pod "7c1d3714-9b66-41db-be48-ec8495162ff3" (UID: "7c1d3714-9b66-41db-be48-ec8495162ff3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:10.856463 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.856426 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1d3714-9b66-41db-be48-ec8495162ff3-istiod-ca-cert" pod "7c1d3714-9b66-41db-be48-ec8495162ff3" (UID: "7c1d3714-9b66-41db-be48-ec8495162ff3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:10.856643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.856512 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:10.856643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.856551 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:10.856643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.856566 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:10.856643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.856581 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:10.859151 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.859093 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-token" pod "7c1d3714-9b66-41db-be48-ec8495162ff3" (UID: "7c1d3714-9b66-41db-be48-ec8495162ff3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:10.864830 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.864801 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-podinfo" pod "7c1d3714-9b66-41db-be48-ec8495162ff3" (UID: "7c1d3714-9b66-41db-be48-ec8495162ff3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:21:10.864950 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.864829 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-kube-api-access-d9r4f" pod "7c1d3714-9b66-41db-be48-ec8495162ff3" (UID: "7c1d3714-9b66-41db-be48-ec8495162ff3"). InnerVolumeSpecName "kube-api-access-d9r4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:10.869672 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.869642 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-envoy" pod "7c1d3714-9b66-41db-be48-ec8495162ff3" (UID: "7c1d3714-9b66-41db-be48-ec8495162ff3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:10.958784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.958695 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1d3714-9b66-41db-be48-ec8495162ff3-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:10.958784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.958728 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:10.958784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.958743 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:10.958784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.958755 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9r4f\" (UniqueName: \"kubernetes.io/projected/7c1d3714-9b66-41db-be48-ec8495162ff3-kube-api-access-d9r4f\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:10.958784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:10.958766 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1d3714-9b66-41db-be48-ec8495162ff3-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.210925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.210861 5399 generic.go:356] "Generic (PLEG): container finished" podID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerID="92716e51a8ef9c94abe72a607c23245bcbae88c074edb177c9544283fe8e08e4" exitCode=0 Nov 02 20:21:11.211044 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.211032 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" event={"ID":"fa98bd03-1a45-496f-a703-2ed86e9b62e9","Type":"ContainerDied","Data":"92716e51a8ef9c94abe72a607c23245bcbae88c074edb177c9544283fe8e08e4"} Nov 02 20:21:11.213135 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.213105 5399 generic.go:356] "Generic (PLEG): container finished" podID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerID="c63567174de3f5f24853e05c5c3a982589e84a8612bf453cb0ee9bfba7a09071" exitCode=0 Nov 02 20:21:11.213363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.213333 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" event={"ID":"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d","Type":"ContainerDied","Data":"c63567174de3f5f24853e05c5c3a982589e84a8612bf453cb0ee9bfba7a09071"} Nov 02 20:21:11.215644 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.215612 5399 generic.go:356] "Generic (PLEG): container finished" podID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerID="a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913" exitCode=0 Nov 02 20:21:11.215805 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.215646 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" event={"ID":"7c1d3714-9b66-41db-be48-ec8495162ff3","Type":"ContainerDied","Data":"a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913"} Nov 02 20:21:11.215805 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.215669 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" Nov 02 20:21:11.215805 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.215719 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2" event={"ID":"7c1d3714-9b66-41db-be48-ec8495162ff3","Type":"ContainerDied","Data":"6b7f896e43dbc985d93b6a135d6a155f30986ea8fd17d14be83edeb00405716d"} Nov 02 20:21:11.222890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.222862 5399 generic.go:356] "Generic (PLEG): container finished" podID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerID="cd997f825b4682dd93187ab4872aebf0127904aeba014c9eefa79089428fdfdf" exitCode=0 Nov 02 20:21:11.223005 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.222910 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" event={"ID":"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5","Type":"ContainerDied","Data":"cd997f825b4682dd93187ab4872aebf0127904aeba014c9eefa79089428fdfdf"} Nov 02 20:21:11.226230 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.226209 5399 scope.go:122] "RemoveContainer" containerID="a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913" Nov 02 20:21:11.254679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.254652 5399 scope.go:122] "RemoveContainer" containerID="a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913" Nov 02 20:21:11.255099 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:21:11.255073 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913\": container with ID starting with a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913 not found: ID does not exist" containerID="a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913" Nov 02 20:21:11.255198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.255110 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913"} err="failed to get container status \"a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913\": rpc error: code = NotFound desc = could not find container \"a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913\": container with ID starting with a2aa26b7a0823e3fcfc52d7c4dbe5e580a288cbbf7e8995358220f2d08de2913 not found: ID does not exist" Nov 02 20:21:11.405004 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.395477 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2"] Nov 02 20:21:11.419193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.419162 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-s64w2"] Nov 02 20:21:11.460393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.460054 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:21:11.518480 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.518453 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:21:11.521461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.521435 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" path="/var/lib/kubelet/pods/7c1d3714-9b66-41db-be48-ec8495162ff3/volumes" Nov 02 20:21:11.561590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.561559 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:21:11.568664 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.568632 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-token\" (UniqueName: \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-token\") pod \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " Nov 02 20:21:11.568838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.568717 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-envoy\") pod \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " Nov 02 20:21:11.568838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.568789 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istiod-ca-cert\") pod \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " Nov 02 20:21:11.568838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.568821 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-certs\") pod \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " Nov 02 20:21:11.569007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.568851 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-podinfo\") pod \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " Nov 02 20:21:11.569007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.568979 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-data\") pod \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " Nov 02 20:21:11.569115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.569064 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-kube-api-access-xs7b7\" (UniqueName: \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-kube-api-access-xs7b7\") pod \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " Nov 02 20:21:11.569115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.569099 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-socket\") pod \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " Nov 02 20:21:11.569245 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.569169 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-credential-socket\") pod \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\" (UID: \"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d\") " Nov 02 20:21:11.569245 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.569214 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istiod-ca-cert" pod "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" (UID: "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:11.569702 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.569678 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-data" pod "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" (UID: "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.570138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.570115 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.570325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.570309 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.570674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.570651 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-socket" pod "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" (UID: "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.570874 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.570855 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-certs" pod "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" (UID: "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.570976 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.570904 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-credential-socket" pod "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" (UID: "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.571485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.571449 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-token" pod "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" (UID: "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:11.572787 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.572765 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-kube-api-access-xs7b7" pod "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" (UID: "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d"). InnerVolumeSpecName "kube-api-access-xs7b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:11.572963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.572941 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-podinfo" pod "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" (UID: "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:21:11.573261 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.573238 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-envoy" pod "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" (UID: "9a7abb95-84f1-4a34-a6b3-403a5d02bd1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.671665 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.671633 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-data\") pod \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " Nov 02 20:21:11.671665 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.671670 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istiod-ca-cert\") pod \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " Nov 02 20:21:11.671953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.671694 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-kube-api-access-gkx2x\" (UniqueName: \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-kube-api-access-gkx2x\") pod \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " Nov 02 20:21:11.671953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.671720 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-token\" (UniqueName: \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-token\") pod \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " Nov 02 20:21:11.671953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.671754 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-envoy\") pod \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " Nov 02 20:21:11.671953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.671820 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-podinfo\") pod \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " Nov 02 20:21:11.671953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.671887 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-socket\") pod \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " Nov 02 20:21:11.671953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.671946 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-data\") pod \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " Nov 02 20:21:11.672289 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.671973 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-token\" (UniqueName: \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-token\") pod \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " Nov 02 20:21:11.672289 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672010 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-socket\") pod \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " Nov 02 20:21:11.672289 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672022 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-data" pod "fa98bd03-1a45-496f-a703-2ed86e9b62e9" (UID: "fa98bd03-1a45-496f-a703-2ed86e9b62e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.672289 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672081 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istiod-ca-cert\") pod \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " Nov 02 20:21:11.672289 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672160 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-envoy\") pod \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " Nov 02 20:21:11.672289 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672248 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-podinfo\") pod \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " Nov 02 20:21:11.672289 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672256 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-socket" pod "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" (UID: "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.672662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672292 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istiod-ca-cert" pod "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" (UID: "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:11.672662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672308 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-certs\") pod \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " Nov 02 20:21:11.672662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672342 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-certs\") pod \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " Nov 02 20:21:11.672662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672387 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-credential-socket\") pod \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\" (UID: \"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5\") " Nov 02 20:21:11.672662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672414 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-kube-api-access-n4cdb\" (UniqueName: \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-kube-api-access-n4cdb\") pod \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " Nov 02 20:21:11.672662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672423 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-data" pod "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" (UID: "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.672662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672477 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-credential-socket\") pod \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\" (UID: \"fa98bd03-1a45-496f-a703-2ed86e9b62e9\") " Nov 02 20:21:11.672662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672604 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istiod-ca-cert" pod "fa98bd03-1a45-496f-a703-2ed86e9b62e9" (UID: "fa98bd03-1a45-496f-a703-2ed86e9b62e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:11.672662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672614 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-socket" pod "fa98bd03-1a45-496f-a703-2ed86e9b62e9" (UID: "fa98bd03-1a45-496f-a703-2ed86e9b62e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.673209 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.672862 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-certs" pod "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" (UID: "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.673353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673328 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673362 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673378 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673394 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673409 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673425 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673440 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673457 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673472 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673488 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs7b7\" (UniqueName: \"kubernetes.io/projected/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-kube-api-access-xs7b7\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673503 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673520 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673534 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.673745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673549 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.674140 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673790 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-certs" pod "fa98bd03-1a45-496f-a703-2ed86e9b62e9" (UID: "fa98bd03-1a45-496f-a703-2ed86e9b62e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.674140 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.673968 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-credential-socket" pod "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" (UID: "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.674986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.674684 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-podinfo" pod "fa98bd03-1a45-496f-a703-2ed86e9b62e9" (UID: "fa98bd03-1a45-496f-a703-2ed86e9b62e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:21:11.674986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.674931 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-envoy" pod "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" (UID: "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.674986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.674942 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-credential-socket" pod "fa98bd03-1a45-496f-a703-2ed86e9b62e9" (UID: "fa98bd03-1a45-496f-a703-2ed86e9b62e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.675221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.675182 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-podinfo" pod "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" (UID: "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:21:11.676004 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.675971 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-kube-api-access-n4cdb" pod "fa98bd03-1a45-496f-a703-2ed86e9b62e9" (UID: "fa98bd03-1a45-496f-a703-2ed86e9b62e9"). InnerVolumeSpecName "kube-api-access-n4cdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:11.676386 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.676363 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-token" pod "fa98bd03-1a45-496f-a703-2ed86e9b62e9" (UID: "fa98bd03-1a45-496f-a703-2ed86e9b62e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:11.677075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.677033 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-token" pod "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" (UID: "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:11.677189 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.677136 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-envoy" pod "fa98bd03-1a45-496f-a703-2ed86e9b62e9" (UID: "fa98bd03-1a45-496f-a703-2ed86e9b62e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:21:11.677910 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.677882 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-kube-api-access-gkx2x" pod "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" (UID: "6f04fe12-86a0-479c-b0c0-2c9df43f3fc5"). InnerVolumeSpecName "kube-api-access-gkx2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:11.775376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775268 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.775376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775327 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.775376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775337 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.775376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775348 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.775376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775362 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4cdb\" (UniqueName: \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-kube-api-access-n4cdb\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.775376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775376 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.775688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775385 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkx2x\" (UniqueName: \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-kube-api-access-gkx2x\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.775688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775394 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.775688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775403 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.775688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775412 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa98bd03-1a45-496f-a703-2ed86e9b62e9-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:11.775688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:11.775420 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:12.232819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.232787 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" Nov 02 20:21:12.233323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.232820 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn" event={"ID":"6f04fe12-86a0-479c-b0c0-2c9df43f3fc5","Type":"ContainerDied","Data":"64d9fb4522ab49c0ca8436a093307f984c9c1bd9d3324065b8d38caeee1ab195"} Nov 02 20:21:12.233323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.232879 5399 scope.go:122] "RemoveContainer" containerID="cd997f825b4682dd93187ab4872aebf0127904aeba014c9eefa79089428fdfdf" Nov 02 20:21:12.234459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.234327 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" Nov 02 20:21:12.234459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.234381 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww" event={"ID":"fa98bd03-1a45-496f-a703-2ed86e9b62e9","Type":"ContainerDied","Data":"7542ba3b57901fa76d014da68383947ef91a1ef5316243da85c29efd9c23827b"} Nov 02 20:21:12.236024 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.235964 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" event={"ID":"9a7abb95-84f1-4a34-a6b3-403a5d02bd1d","Type":"ContainerDied","Data":"7cf7c593d9803c504a3f90e80c3e3de41f61526a0a2e17ef312d21374a2e088a"} Nov 02 20:21:12.236024 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.235991 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr" Nov 02 20:21:12.245327 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.245225 5399 scope.go:122] "RemoveContainer" containerID="92716e51a8ef9c94abe72a607c23245bcbae88c074edb177c9544283fe8e08e4" Nov 02 20:21:12.255590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.255568 5399 scope.go:122] "RemoveContainer" containerID="c63567174de3f5f24853e05c5c3a982589e84a8612bf453cb0ee9bfba7a09071" Nov 02 20:21:12.394713 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.394683 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr"] Nov 02 20:21:12.400228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.400197 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-k5nrr"] Nov 02 20:21:12.500661 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.500578 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww"] Nov 02 20:21:12.504089 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.504063 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-splww"] Nov 02 20:21:12.603369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.603335 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn"] Nov 02 20:21:12.606080 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:12.606054 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-lgncn"] Nov 02 20:21:13.511251 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:13.511217 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" path="/var/lib/kubelet/pods/6f04fe12-86a0-479c-b0c0-2c9df43f3fc5/volumes" Nov 02 20:21:13.511763 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:13.511746 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" path="/var/lib/kubelet/pods/9a7abb95-84f1-4a34-a6b3-403a5d02bd1d/volumes" Nov 02 20:21:13.512180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:13.512168 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" path="/var/lib/kubelet/pods/fa98bd03-1a45-496f-a703-2ed86e9b62e9/volumes" Nov 02 20:21:16.177573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:16.177542 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:21:18.237866 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:21:18.237786 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:21:38.203785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:38.203753 5399 scope.go:122] "RemoveContainer" containerID="1a925c2a41d5e10381ad8bc9127b13ad172b69a6bb8cdac1b61f90553bed9a8d" Nov 02 20:21:38.214375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:38.214350 5399 scope.go:122] "RemoveContainer" containerID="d6051846cec41b56589cefe069a7fc4bd551aee8b04a99d00f92d011e7712fd1" Nov 02 20:21:38.223853 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:21:38.223829 5399 scope.go:122] "RemoveContainer" containerID="83e1a6fe19f02f5b8abbc4bf553b5d6570398a8b400b570ce086aa3dd4bc602b" Nov 02 20:21:48.236965 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:21:48.236923 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:22:06.456230 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:06.456180 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:22:10.379186 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:10.379139 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Nov 02 20:22:10.466916 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:10.466190 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:22:10.495124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:10.494805 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:22:10.628344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:10.626392 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:22:10.669518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:10.669393 5399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Nov 02 20:22:15.970485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:15.970439 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:22:15.970998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:15.970529 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:22:16.927544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:16.927457 5399 generic.go:356] "Generic (PLEG): container finished" podID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerID="703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8" exitCode=0 Nov 02 20:22:16.927771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:16.927533 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" event={"ID":"fe65bde2-a8b8-472b-bc2f-4a518e2702de","Type":"ContainerDied","Data":"703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8"} Nov 02 20:22:16.928043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:16.928026 5399 scope.go:122] "RemoveContainer" containerID="703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8" Nov 02 20:22:16.929684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:16.929653 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:16.929773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:16.929725 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:16.929773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:16.929757 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:17.956193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:17.956154 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" event={"ID":"fe65bde2-a8b8-472b-bc2f-4a518e2702de","Type":"ContainerStarted","Data":"ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5"} Nov 02 20:22:18.263812 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:22:18.263722 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:22:18.386660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:18.386628 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:22:18.386818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:18.386724 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 20:22:18.386818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:18.386766 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 20:22:19.386487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:19.386456 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 20:22:19.386974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:19.386507 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 20:22:20.386825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:20.386784 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 20:22:20.387230 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:20.386852 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 20:22:21.386524 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:21.386487 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 20:22:21.386755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:21.386561 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 20:22:22.386381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:22.386344 5399 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 02 20:22:22.386847 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:22.386404 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 02 20:22:23.390711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:23.390681 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:22:23.391227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:23.391063 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:22:23.391819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:23.391797 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:22:48.256076 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:22:48.255981 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:22:51.030015 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:22:51.029979 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:23:18.237230 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:23:18.237187 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:23:21.042408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:21.042370 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98"] Nov 02 20:23:21.042877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:21.042570 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" containerID="cri-o://585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147" gracePeriod=30 Nov 02 20:23:21.088702 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:21.088656 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s"] Nov 02 20:23:21.088920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:21.088878 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" containerID="cri-o://7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8" gracePeriod=30 Nov 02 20:23:23.299795 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:23.299754 5399 patch_prober.go:29] interesting pod/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:23:23.300242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:23.299815 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:23:25.919988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.919951 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm"] Nov 02 20:23:25.920631 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920617 5399 cpu_manager.go:414] "containerMap: removing container" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" Nov 02 20:23:25.920679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920632 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" Nov 02 20:23:25.920679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920662 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" Nov 02 20:23:25.920679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920667 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" Nov 02 20:23:25.920679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920677 5399 cpu_manager.go:414] "containerMap: removing container" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" Nov 02 20:23:25.920808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920683 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" Nov 02 20:23:25.920808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920701 5399 cpu_manager.go:414] "containerMap: removing container" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerName="istio-proxy" Nov 02 20:23:25.920808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920706 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerName="istio-proxy" Nov 02 20:23:25.920897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920812 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7abb95-84f1-4a34-a6b3-403a5d02bd1d" containerName="istio-proxy" Nov 02 20:23:25.920897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920823 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1d3714-9b66-41db-be48-ec8495162ff3" containerName="istio-proxy" Nov 02 20:23:25.920897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920830 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f04fe12-86a0-479c-b0c0-2c9df43f3fc5" containerName="istio-proxy" Nov 02 20:23:25.920897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.920839 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa98bd03-1a45-496f-a703-2ed86e9b62e9" containerName="istio-proxy" Nov 02 20:23:25.924371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.924342 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:25.977229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:25.977194 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm"] Nov 02 20:23:26.018183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.018146 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:23:26.018377 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.018242 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:23:26.095734 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.095700 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575830cb-4csqm\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:26.095892 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.095763 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575830cb-4csqm\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:26.095996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.095980 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwm5j\" (UniqueName: \"kubernetes.io/projected/f4898325-a1c0-4353-b407-4cc188f2f963-kube-api-access-gwm5j\") pod \"maas-api-revoke-sub-575830cb-4csqm\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:26.197715 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.197633 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575830cb-4csqm\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:26.197715 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.197695 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575830cb-4csqm\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:26.197938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.197858 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwm5j\" (UniqueName: \"kubernetes.io/projected/f4898325-a1c0-4353-b407-4cc188f2f963-kube-api-access-gwm5j\") pod \"maas-api-revoke-sub-575830cb-4csqm\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:26.198575 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.198549 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575830cb-4csqm\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:26.200419 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.200395 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575830cb-4csqm\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:26.253604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.253569 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwm5j\" (UniqueName: \"kubernetes.io/projected/f4898325-a1c0-4353-b407-4cc188f2f963-kube-api-access-gwm5j\") pod \"maas-api-revoke-sub-575830cb-4csqm\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:26.406334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.406307 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:23:26.413900 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.413875 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:23:26.474792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.474709 5399 generic.go:356] "Generic (PLEG): container finished" podID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerID="585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147" exitCode=0 Nov 02 20:23:26.474792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.474757 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" Nov 02 20:23:26.474996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.474791 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" event={"ID":"5498af7b-e63c-4dee-ac23-6e48f373243e","Type":"ContainerDied","Data":"585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147"} Nov 02 20:23:26.474996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.474892 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98" event={"ID":"5498af7b-e63c-4dee-ac23-6e48f373243e","Type":"ContainerDied","Data":"e9fe419ca147d841ae9b47f683f3f96e2067a6a65f8566e7cc58a68456a5bc6b"} Nov 02 20:23:26.474996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.474927 5399 scope.go:122] "RemoveContainer" containerID="585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147" Nov 02 20:23:26.476164 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.476144 5399 generic.go:356] "Generic (PLEG): container finished" podID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerID="7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8" exitCode=0 Nov 02 20:23:26.476262 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.476193 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" Nov 02 20:23:26.476262 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.476234 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" event={"ID":"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22","Type":"ContainerDied","Data":"7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8"} Nov 02 20:23:26.476395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.476322 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s" event={"ID":"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22","Type":"ContainerDied","Data":"b2ae9875fca301f42f5a3cff146838b50c6365338b41842698d7b55ae12d7953"} Nov 02 20:23:26.487738 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.487717 5399 scope.go:122] "RemoveContainer" containerID="585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147" Nov 02 20:23:26.488083 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:23:26.488057 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147\": container with ID starting with 585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147 not found: ID does not exist" containerID="585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147" Nov 02 20:23:26.488153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.488097 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147"} err="failed to get container status \"585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147\": rpc error: code = NotFound desc = could not find container \"585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147\": container with ID starting with 585593d9ae1f463834e7dfbef4ae17def89727899bd692cc345eb6d548466147 not found: ID does not exist" Nov 02 20:23:26.488153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.488121 5399 scope.go:122] "RemoveContainer" containerID="7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8" Nov 02 20:23:26.499189 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.499172 5399 scope.go:122] "RemoveContainer" containerID="7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8" Nov 02 20:23:26.499514 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:23:26.499493 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8\": container with ID starting with 7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8 not found: ID does not exist" containerID="7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8" Nov 02 20:23:26.499594 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.499525 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8"} err="failed to get container status \"7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8\": rpc error: code = NotFound desc = could not find container \"7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8\": container with ID starting with 7eb393890e2de25c9b32542f3c80c1521052b6408d829e3bd615c040dbe748e8 not found: ID does not exist" Nov 02 20:23:26.505865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.505842 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-data\") pod \"5498af7b-e63c-4dee-ac23-6e48f373243e\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " Nov 02 20:23:26.505988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.505884 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-podinfo\") pod \"5498af7b-e63c-4dee-ac23-6e48f373243e\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " Nov 02 20:23:26.505988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.505913 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5498af7b-e63c-4dee-ac23-6e48f373243e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5498af7b-e63c-4dee-ac23-6e48f373243e-istiod-ca-cert\") pod \"5498af7b-e63c-4dee-ac23-6e48f373243e\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " Nov 02 20:23:26.506094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506009 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-socket\") pod \"5498af7b-e63c-4dee-ac23-6e48f373243e\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " Nov 02 20:23:26.506094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506051 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-credential-socket\") pod \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " Nov 02 20:23:26.506094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506076 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-data" pod "5498af7b-e63c-4dee-ac23-6e48f373243e" (UID: "5498af7b-e63c-4dee-ac23-6e48f373243e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:26.506209 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506100 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-certs\") pod \"5498af7b-e63c-4dee-ac23-6e48f373243e\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " Nov 02 20:23:26.506249 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506218 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-token\" (UniqueName: \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-token\") pod \"5498af7b-e63c-4dee-ac23-6e48f373243e\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " Nov 02 20:23:26.506323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506218 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-credential-socket" pod "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" (UID: "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:26.506323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506226 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-socket" pod "5498af7b-e63c-4dee-ac23-6e48f373243e" (UID: "5498af7b-e63c-4dee-ac23-6e48f373243e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:26.506401 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506264 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5498af7b-e63c-4dee-ac23-6e48f373243e-istiod-ca-cert" pod "5498af7b-e63c-4dee-ac23-6e48f373243e" (UID: "5498af7b-e63c-4dee-ac23-6e48f373243e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:26.506463 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506437 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-certs" pod "5498af7b-e63c-4dee-ac23-6e48f373243e" (UID: "5498af7b-e63c-4dee-ac23-6e48f373243e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:26.506523 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506505 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-podinfo\") pod \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " Nov 02 20:23:26.506572 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506541 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-kube-api-access-gg6xt\" (UniqueName: \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-kube-api-access-gg6xt\") pod \"5498af7b-e63c-4dee-ac23-6e48f373243e\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " Nov 02 20:23:26.506614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506574 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-credential-socket\") pod \"5498af7b-e63c-4dee-ac23-6e48f373243e\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " Nov 02 20:23:26.506614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506596 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-envoy\") pod \"5498af7b-e63c-4dee-ac23-6e48f373243e\" (UID: \"5498af7b-e63c-4dee-ac23-6e48f373243e\") " Nov 02 20:23:26.506684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506644 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-socket\") pod \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " Nov 02 20:23:26.506684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506671 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-kube-api-access-xnfbw\" (UniqueName: \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-kube-api-access-xnfbw\") pod \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " Nov 02 20:23:26.506753 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506739 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-certs\") pod \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " Nov 02 20:23:26.506808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506792 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istiod-ca-cert\") pod \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " Nov 02 20:23:26.506864 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506850 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-data\") pod \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " Nov 02 20:23:26.506929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506876 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-token\" (UniqueName: \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-token\") pod \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " Nov 02 20:23:26.506929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.506914 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-envoy\") pod \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\" (UID: \"9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22\") " Nov 02 20:23:26.507058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507035 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-socket" pod "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" (UID: "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:26.507443 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507216 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-credential-socket" pod "5498af7b-e63c-4dee-ac23-6e48f373243e" (UID: "5498af7b-e63c-4dee-ac23-6e48f373243e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:26.507756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507558 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-data" pod "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" (UID: "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:26.507756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507617 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.507756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507640 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5498af7b-e63c-4dee-ac23-6e48f373243e-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.507756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507658 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.507756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507673 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.507756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507688 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.507756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507703 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.507756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507718 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.508230 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.507758 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-certs" pod "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" (UID: "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:26.508369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.508332 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istiod-ca-cert" pod "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" (UID: "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:26.508445 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.508419 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-podinfo" pod "5498af7b-e63c-4dee-ac23-6e48f373243e" (UID: "5498af7b-e63c-4dee-ac23-6e48f373243e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:23:26.508884 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.508857 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-kube-api-access-gg6xt" pod "5498af7b-e63c-4dee-ac23-6e48f373243e" (UID: "5498af7b-e63c-4dee-ac23-6e48f373243e"). InnerVolumeSpecName "kube-api-access-gg6xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:26.509620 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.509595 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-token" pod "5498af7b-e63c-4dee-ac23-6e48f373243e" (UID: "5498af7b-e63c-4dee-ac23-6e48f373243e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:26.509701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.509671 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-podinfo" pod "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" (UID: "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:23:26.509794 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.509771 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-envoy" pod "5498af7b-e63c-4dee-ac23-6e48f373243e" (UID: "5498af7b-e63c-4dee-ac23-6e48f373243e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:26.509896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.509881 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-envoy" pod "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" (UID: "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:26.510088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.510075 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-kube-api-access-xnfbw" pod "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" (UID: "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22"). InnerVolumeSpecName "kube-api-access-xnfbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:26.510883 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.510862 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-token" pod "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" (UID: "9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:26.538720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.538689 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:26.612648 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612596 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.612808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612656 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnfbw\" (UniqueName: \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-kube-api-access-xnfbw\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.612808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612698 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.612808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612714 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.612808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612729 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.612808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612744 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.612808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612760 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.612808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612776 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.612808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612791 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.612808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612805 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.613267 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.612820 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg6xt\" (UniqueName: \"kubernetes.io/projected/5498af7b-e63c-4dee-ac23-6e48f373243e-kube-api-access-gg6xt\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:26.898868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.898829 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98"] Nov 02 20:23:26.902560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.902530 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-26k98"] Nov 02 20:23:26.908407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:26.908384 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm"] Nov 02 20:23:26.909079 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:23:26.909058 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4898325_a1c0_4353_b407_4cc188f2f963.slice/crio-9ccd79b36ce7a8eb429b37b19f63cf33634ca18694b1a1d88a467cd03b82ab1f WatchSource:0}: Error finding container 9ccd79b36ce7a8eb429b37b19f63cf33634ca18694b1a1d88a467cd03b82ab1f: Status 404 returned error can't find the container with id 9ccd79b36ce7a8eb429b37b19f63cf33634ca18694b1a1d88a467cd03b82ab1f Nov 02 20:23:27.005688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:27.005652 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s"] Nov 02 20:23:27.009086 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:27.009060 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ae8e6e-openshift-default-7c8c89778c-q2z8s"] Nov 02 20:23:27.488440 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:27.488342 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" event={"ID":"f4898325-a1c0-4353-b407-4cc188f2f963","Type":"ContainerStarted","Data":"9adce58c47efecb7c2d0404e751876a622beb59e5025e0804a215fec340f8cf2"} Nov 02 20:23:27.488440 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:27.488407 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" event={"ID":"f4898325-a1c0-4353-b407-4cc188f2f963","Type":"ContainerStarted","Data":"9ccd79b36ce7a8eb429b37b19f63cf33634ca18694b1a1d88a467cd03b82ab1f"} Nov 02 20:23:27.509681 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:27.509645 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" path="/var/lib/kubelet/pods/5498af7b-e63c-4dee-ac23-6e48f373243e/volumes" Nov 02 20:23:27.510334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:27.510314 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" path="/var/lib/kubelet/pods/9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22/volumes" Nov 02 20:23:27.599094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:27.599029 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" podStartSLOduration=2.5990117440000002 podStartE2EDuration="2.599011744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:27.59342999 +0000 UTC m=+1850.204082095" watchObservedRunningTime="2026-11-02 20:23:27.599011744 +0000 UTC m=+1850.209663836" Nov 02 20:23:28.502999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:28.502966 5399 generic.go:356] "Generic (PLEG): container finished" podID="f4898325-a1c0-4353-b407-4cc188f2f963" containerID="9adce58c47efecb7c2d0404e751876a622beb59e5025e0804a215fec340f8cf2" exitCode=0 Nov 02 20:23:28.503471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:28.503041 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" event={"ID":"f4898325-a1c0-4353-b407-4cc188f2f963","Type":"ContainerDied","Data":"9adce58c47efecb7c2d0404e751876a622beb59e5025e0804a215fec340f8cf2"} Nov 02 20:23:29.681897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:29.681868 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:29.754626 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:29.754591 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-xks-ca\") pod \"f4898325-a1c0-4353-b407-4cc188f2f963\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " Nov 02 20:23:29.754832 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:29.754645 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4898325-a1c0-4353-b407-4cc188f2f963-kube-api-access-gwm5j\" (UniqueName: \"kubernetes.io/projected/f4898325-a1c0-4353-b407-4cc188f2f963-kube-api-access-gwm5j\") pod \"f4898325-a1c0-4353-b407-4cc188f2f963\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " Nov 02 20:23:29.754832 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:29.754760 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-openshift-ca\") pod \"f4898325-a1c0-4353-b407-4cc188f2f963\" (UID: \"f4898325-a1c0-4353-b407-4cc188f2f963\") " Nov 02 20:23:29.755121 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:29.755092 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-openshift-ca" pod "f4898325-a1c0-4353-b407-4cc188f2f963" (UID: "f4898325-a1c0-4353-b407-4cc188f2f963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:29.755296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:29.755263 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:29.756842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:29.756810 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-xks-ca" pod "f4898325-a1c0-4353-b407-4cc188f2f963" (UID: "f4898325-a1c0-4353-b407-4cc188f2f963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:29.756935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:29.756887 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4898325-a1c0-4353-b407-4cc188f2f963-kube-api-access-gwm5j" pod "f4898325-a1c0-4353-b407-4cc188f2f963" (UID: "f4898325-a1c0-4353-b407-4cc188f2f963"). InnerVolumeSpecName "kube-api-access-gwm5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:29.856645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:29.856608 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4898325-a1c0-4353-b407-4cc188f2f963-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:29.856645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:29.856641 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwm5j\" (UniqueName: \"kubernetes.io/projected/f4898325-a1c0-4353-b407-4cc188f2f963-kube-api-access-gwm5j\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:30.517704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:30.517673 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" Nov 02 20:23:30.517896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:30.517720 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm" event={"ID":"f4898325-a1c0-4353-b407-4cc188f2f963","Type":"ContainerDied","Data":"9ccd79b36ce7a8eb429b37b19f63cf33634ca18694b1a1d88a467cd03b82ab1f"} Nov 02 20:23:30.517896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:30.517774 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ccd79b36ce7a8eb429b37b19f63cf33634ca18694b1a1d88a467cd03b82ab1f" Nov 02 20:23:33.909142 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.909102 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck"] Nov 02 20:23:33.909720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.909706 5399 cpu_manager.go:414] "containerMap: removing container" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" Nov 02 20:23:33.909778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.909721 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" Nov 02 20:23:33.909778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.909749 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f4898325-a1c0-4353-b407-4cc188f2f963" containerName="revoke-keys" Nov 02 20:23:33.909778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.909754 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4898325-a1c0-4353-b407-4cc188f2f963" containerName="revoke-keys" Nov 02 20:23:33.909778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.909769 5399 cpu_manager.go:414] "containerMap: removing container" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" Nov 02 20:23:33.909778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.909774 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" Nov 02 20:23:33.909920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.909867 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6ae9d0-3900-4e9b-ac3b-a1ccb9f1da22" containerName="istio-proxy" Nov 02 20:23:33.909920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.909880 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4898325-a1c0-4353-b407-4cc188f2f963" containerName="revoke-keys" Nov 02 20:23:33.909920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.909888 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5498af7b-e63c-4dee-ac23-6e48f373243e" containerName="istio-proxy" Nov 02 20:23:33.913674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.913657 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:33.966401 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:33.966360 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck"] Nov 02 20:23:34.008420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.008392 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7dcb03-serving-cert\"" Nov 02 20:23:34.014450 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.014419 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46tg\" (UniqueName: \"kubernetes.io/projected/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-kube-api-access-f46tg\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.014537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.014504 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.014656 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.014632 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-metrics-certs\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.014701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.014672 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.115872 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.115839 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46tg\" (UniqueName: \"kubernetes.io/projected/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-kube-api-access-f46tg\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.116064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.115903 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.116064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.115975 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-metrics-certs\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.116064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.116011 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.116297 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:23:34.116255 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-serving-cert: secret "maas-api-e2e-inf-a-7dcb03-serving-cert" not found Nov 02 20:23:34.116367 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:23:34.116350 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-maas-api-tls podName:dbb75c5d-da33-49c5-aeda-03a3e0e74a5f nodeName:}" failed. No retries permitted until 2026-11-02 20:23:34.616328835 +0000 UTC m=+1857.226980904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-maas-api-tls") pod "maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" (UID: "dbb75c5d-da33-49c5-aeda-03a3e0e74a5f") : secret "maas-api-e2e-inf-a-7dcb03-serving-cert" not found Nov 02 20:23:34.116700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.116681 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.118622 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.118605 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-metrics-certs\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.171537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.171462 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46tg\" (UniqueName: \"kubernetes.io/projected/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-kube-api-access-f46tg\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.623962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.623921 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.626432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.626408 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:34.827823 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:34.827790 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:35.198009 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:35.197979 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck"] Nov 02 20:23:35.198413 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:23:35.198314 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb75c5d_da33_49c5_aeda_03a3e0e74a5f.slice/crio-7301a84057c8d933fe6b7b32d763e977506b96539157d05ddb236e946991d616 WatchSource:0}: Error finding container 7301a84057c8d933fe6b7b32d763e977506b96539157d05ddb236e946991d616: Status 404 returned error can't find the container with id 7301a84057c8d933fe6b7b32d763e977506b96539157d05ddb236e946991d616 Nov 02 20:23:35.552771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:35.552680 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" event={"ID":"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f","Type":"ContainerStarted","Data":"7301a84057c8d933fe6b7b32d763e977506b96539157d05ddb236e946991d616"} Nov 02 20:23:36.561770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:36.561733 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" event={"ID":"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f","Type":"ContainerStarted","Data":"2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b"} Nov 02 20:23:36.562188 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:36.561844 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:36.668776 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:36.668717 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" podStartSLOduration=2.802502936 podStartE2EDuration="3.668698542s" totalImagesPullingTime="866.195606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:36.668647404 +0000 UTC m=+1859.279299498" watchObservedRunningTime="2026-11-02 20:23:36.668698542 +0000 UTC m=+1859.279350635" Nov 02 20:23:42.574888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:42.574858 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:23:43.001711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.001627 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns"] Nov 02 20:23:43.007488 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.007471 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.055980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.055944 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c76017-serving-cert\"" Nov 02 20:23:43.058781 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.058755 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns"] Nov 02 20:23:43.143858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.143824 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2cp9\" (UniqueName: \"kubernetes.io/projected/a132e255-d892-4039-9a76-936c22c94fe1-kube-api-access-w2cp9\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.144042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.143993 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-metrics-certs\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.144176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.144158 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a132e255-d892-4039-9a76-936c22c94fe1-openshift-service-ca\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.144232 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.144214 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-maas-api-tls\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.245861 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.245777 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-metrics-certs\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.246062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.245912 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a132e255-d892-4039-9a76-936c22c94fe1-openshift-service-ca\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.246062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.245961 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-maas-api-tls\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.246237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.246114 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2cp9\" (UniqueName: \"kubernetes.io/projected/a132e255-d892-4039-9a76-936c22c94fe1-kube-api-access-w2cp9\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.248891 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:23:43.247064 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-serving-cert: secret "maas-api-e2e-api-a-c76017-serving-cert" not found Nov 02 20:23:43.248891 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:23:43.247150 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-maas-api-tls podName:a132e255-d892-4039-9a76-936c22c94fe1 nodeName:}" failed. No retries permitted until 2026-11-02 20:23:43.747128986 +0000 UTC m=+1866.357781074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-maas-api-tls") pod "maas-api-e2e-api-a-c76017-677b8795c-p25ns" (UID: "a132e255-d892-4039-9a76-936c22c94fe1") : secret "maas-api-e2e-api-a-c76017-serving-cert" not found Nov 02 20:23:43.248891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.247650 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a132e255-d892-4039-9a76-936c22c94fe1-openshift-service-ca\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.249798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.249773 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-metrics-certs\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.299571 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.299493 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2cp9\" (UniqueName: \"kubernetes.io/projected/a132e255-d892-4039-9a76-936c22c94fe1-kube-api-access-w2cp9\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.751699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.751660 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-maas-api-tls\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.754256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.754230 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-maas-api-tls\") pod \"maas-api-e2e-api-a-c76017-677b8795c-p25ns\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:43.921342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:43.921270 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:44.293583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:44.293551 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns"] Nov 02 20:23:44.294983 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:23:44.294949 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda132e255_d892_4039_9a76_936c22c94fe1.slice/crio-861ac54e05084b8cb47463169a812939f17bbb752cb4e7165b178658dd685b73 WatchSource:0}: Error finding container 861ac54e05084b8cb47463169a812939f17bbb752cb4e7165b178658dd685b73: Status 404 returned error can't find the container with id 861ac54e05084b8cb47463169a812939f17bbb752cb4e7165b178658dd685b73 Nov 02 20:23:44.621033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:44.620996 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" event={"ID":"a132e255-d892-4039-9a76-936c22c94fe1","Type":"ContainerStarted","Data":"861ac54e05084b8cb47463169a812939f17bbb752cb4e7165b178658dd685b73"} Nov 02 20:23:45.630440 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:45.630398 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" event={"ID":"a132e255-d892-4039-9a76-936c22c94fe1","Type":"ContainerStarted","Data":"b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122"} Nov 02 20:23:45.630864 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:45.630514 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:23:45.740903 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:45.740845 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" podStartSLOduration=2.929938442 podStartE2EDuration="3.740830496s" totalImagesPullingTime="810.892054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:45.736826599 +0000 UTC m=+1868.347478691" watchObservedRunningTime="2026-11-02 20:23:45.740830496 +0000 UTC m=+1868.351482587" Nov 02 20:23:48.237230 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:23:48.237182 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:23:51.643906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:23:51.643869 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:24:06.974027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:06.973989 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:24:15.508567 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.508537 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57"] Nov 02 20:24:15.513075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.513047 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.550344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.550308 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k"] Nov 02 20:24:15.554402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.554384 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.562154 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.562131 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7dcb03-openshift-default-dockercfg-mr2wr\"" Nov 02 20:24:15.581850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.581815 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.581973 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.581864 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.582123 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.582108 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.582248 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.582227 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.582336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.582297 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.582336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.582329 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.582529 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.582507 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc5z9\" (UniqueName: \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-kube-api-access-sc5z9\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.582618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.582599 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f8a8a3-fd48-43d2-bf61-87ed31128081-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.582685 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.582664 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.600975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.600948 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57"] Nov 02 20:24:15.653099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.653061 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k"] Nov 02 20:24:15.684140 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684108 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.684354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684163 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.684354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684198 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.684354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684241 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.684354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684310 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc5z9\" (UniqueName: \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-kube-api-access-sc5z9\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.684354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684349 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f8a8a3-fd48-43d2-bf61-87ed31128081-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.684629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684375 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87528081-027a-4246-8cf4-70dbea59d933-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.684629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684403 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.684629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684427 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.684629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684478 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.684629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684520 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqm68\" (UniqueName: \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-kube-api-access-xqm68\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.684629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684590 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.684870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684661 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.684870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684693 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.684870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684721 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87528081-027a-4246-8cf4-70dbea59d933-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.684870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684752 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.684870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684792 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.685024 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.684938 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.685024 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.685012 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.685291 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.685254 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.686128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.685990 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.686128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.686119 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.686707 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.686683 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f8a8a3-fd48-43d2-bf61-87ed31128081-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.687771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.687749 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.687771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.687763 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.740598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.740570 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc5z9\" (UniqueName: \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-kube-api-access-sc5z9\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.740772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.740562 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.786222 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786138 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786325 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786423 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87528081-027a-4246-8cf4-70dbea59d933-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786456 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786591 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786506 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786588 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786605 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqm68\" (UniqueName: \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-kube-api-access-xqm68\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786680 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786712 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786735 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786749 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.786905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786760 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87528081-027a-4246-8cf4-70dbea59d933-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.787237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.786823 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.787683 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.787659 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87528081-027a-4246-8cf4-70dbea59d933-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.792130 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.789891 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87528081-027a-4246-8cf4-70dbea59d933-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.792130 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.789904 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.827457 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.827428 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:15.839831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.839801 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.839950 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.839915 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqm68\" (UniqueName: \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-kube-api-access-xqm68\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:15.867739 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:15.867700 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:16.247387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.247352 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57"] Nov 02 20:24:16.289444 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:24:16.289410 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f8a8a3_fd48_43d2_bf61_87ed31128081.slice/crio-7a9b9da9bb62087e7b68b5e6918b48b08364e7189919b2f3571118c14dd35e16 WatchSource:0}: Error finding container 7a9b9da9bb62087e7b68b5e6918b48b08364e7189919b2f3571118c14dd35e16: Status 404 returned error can't find the container with id 7a9b9da9bb62087e7b68b5e6918b48b08364e7189919b2f3571118c14dd35e16 Nov 02 20:24:16.292198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.292161 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:16.292330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.292234 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:16.292330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.292267 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:16.301038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.301011 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k"] Nov 02 20:24:16.306174 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:24:16.306149 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87528081_027a_4246_8cf4_70dbea59d933.slice/crio-d687204fadc7abf299ecf023d135435fbd67feaaa41cf346400beb5f8ce9fcfb WatchSource:0}: Error finding container d687204fadc7abf299ecf023d135435fbd67feaaa41cf346400beb5f8ce9fcfb: Status 404 returned error can't find the container with id d687204fadc7abf299ecf023d135435fbd67feaaa41cf346400beb5f8ce9fcfb Nov 02 20:24:16.309112 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.309080 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:16.309216 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.309151 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:16.309216 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.309195 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:16.885985 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.885894 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" event={"ID":"71f8a8a3-fd48-43d2-bf61-87ed31128081","Type":"ContainerStarted","Data":"4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85"} Nov 02 20:24:16.886860 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.886827 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" event={"ID":"71f8a8a3-fd48-43d2-bf61-87ed31128081","Type":"ContainerStarted","Data":"7a9b9da9bb62087e7b68b5e6918b48b08364e7189919b2f3571118c14dd35e16"} Nov 02 20:24:16.888877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.888849 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" event={"ID":"87528081-027a-4246-8cf4-70dbea59d933","Type":"ContainerStarted","Data":"e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36"} Nov 02 20:24:16.889016 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:16.888902 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" event={"ID":"87528081-027a-4246-8cf4-70dbea59d933","Type":"ContainerStarted","Data":"d687204fadc7abf299ecf023d135435fbd67feaaa41cf346400beb5f8ce9fcfb"} Nov 02 20:24:17.006956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:17.006894 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" podStartSLOduration=2.006879767 podStartE2EDuration="2.006879767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:17.004110688 +0000 UTC m=+1899.614762779" watchObservedRunningTime="2026-11-02 20:24:17.006879767 +0000 UTC m=+1899.617531857" Nov 02 20:24:17.117884 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:17.117829 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" podStartSLOduration=2.117814548 podStartE2EDuration="2.117814548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:17.114890987 +0000 UTC m=+1899.725543078" watchObservedRunningTime="2026-11-02 20:24:17.117814548 +0000 UTC m=+1899.728466639" Nov 02 20:24:17.875406 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:17.861369 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:17.875406 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:17.861521 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 20:24:17.875406 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:17.861562 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" podUID="71f8a8a3-fd48-43d2-bf61-87ed31128081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 20:24:17.875406 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:17.874647 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:17.876901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:17.876868 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:24:17.877096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:17.877068 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" podUID="87528081-027a-4246-8cf4-70dbea59d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:24:18.121303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.121262 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs"] Nov 02 20:24:18.125578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.125515 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.189121 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.189088 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs"] Nov 02 20:24:18.277301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.267257 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c76017-openshift-default-dockercfg-p9q56\"" Nov 02 20:24:18.291313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.290867 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.291313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.290922 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.291313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.290972 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2e6659-eb29-4b1e-b1e9-17b25342836d-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.291313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.291017 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.291313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.291075 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.291313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.291097 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.291313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.291132 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.291313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.291160 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.291313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.291184 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr4s9\" (UniqueName: \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-kube-api-access-zr4s9\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.297645 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:24:18.297613 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:24:18.392423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.392335 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr4s9\" (UniqueName: \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-kube-api-access-zr4s9\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.392423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.392414 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.392617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.392463 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.392617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.392521 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2e6659-eb29-4b1e-b1e9-17b25342836d-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.392617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.392580 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.392749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.392646 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.392749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.392668 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.392749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.392702 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.392749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.392731 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.393350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.393328 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.393524 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.393504 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2e6659-eb29-4b1e-b1e9-17b25342836d-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.393753 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.393734 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.393753 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.393742 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.394097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.394000 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.395683 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.395653 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.401134 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.400992 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.461354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.454191 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.470211 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.470177 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr4s9\" (UniqueName: \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-kube-api-access-zr4s9\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.492782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.490892 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:18.865300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.853460 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 20:24:18.865300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.853560 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" podUID="71f8a8a3-fd48-43d2-bf61-87ed31128081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 20:24:18.881297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.872909 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:24:18.881297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:18.872970 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" podUID="87528081-027a-4246-8cf4-70dbea59d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:24:19.170303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:19.160831 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs"] Nov 02 20:24:19.181528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:19.181483 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:19.181640 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:19.181584 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:19.181640 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:19.181632 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:19.850299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:19.849548 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 20:24:19.850299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:19.849623 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" podUID="71f8a8a3-fd48-43d2-bf61-87ed31128081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 20:24:19.871298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:19.870704 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:24:19.871298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:19.870782 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" podUID="87528081-027a-4246-8cf4-70dbea59d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:24:19.918388 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:19.918357 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" event={"ID":"df2e6659-eb29-4b1e-b1e9-17b25342836d","Type":"ContainerStarted","Data":"03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910"} Nov 02 20:24:19.918609 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:19.918594 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" event={"ID":"df2e6659-eb29-4b1e-b1e9-17b25342836d","Type":"ContainerStarted","Data":"987cf68aefb28731db92afafbb800977aefb2edf0ccfebf7671fb55354767ba4"} Nov 02 20:24:20.065108 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.050636 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" podStartSLOduration=2.050616257 podStartE2EDuration="2.050616257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:20.043564061 +0000 UTC m=+1902.654216153" watchObservedRunningTime="2026-11-02 20:24:20.050616257 +0000 UTC m=+1902.661268351" Nov 02 20:24:20.094741 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.094709 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f"] Nov 02 20:24:20.099684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.099662 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.148298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.135264 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnhj9\" (UniqueName: \"kubernetes.io/projected/cf408fe6-a44d-479c-9aa6-a9ce068124c5-kube-api-access-wnhj9\") pod \"maas-api-revoke-sub-b62629a1-qlz5f\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.148298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.135470 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b62629a1-qlz5f\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.148298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.135501 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b62629a1-qlz5f\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.165297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.149574 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:24:20.165297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.152093 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:24:20.193334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.190256 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f"] Nov 02 20:24:20.271301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.270261 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b62629a1-qlz5f\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.271301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.270338 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b62629a1-qlz5f\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.271301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.270535 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnhj9\" (UniqueName: \"kubernetes.io/projected/cf408fe6-a44d-479c-9aa6-a9ce068124c5-kube-api-access-wnhj9\") pod \"maas-api-revoke-sub-b62629a1-qlz5f\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.287297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.271825 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b62629a1-qlz5f\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.295355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.294228 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b62629a1-qlz5f\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.324932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.324896 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnhj9\" (UniqueName: \"kubernetes.io/projected/cf408fe6-a44d-479c-9aa6-a9ce068124c5-kube-api-access-wnhj9\") pod \"maas-api-revoke-sub-b62629a1-qlz5f\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.416018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.415944 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:20.493307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.491461 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:20.493916 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.493876 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 20:24:20.494026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.493921 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 20:24:20.829321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.828717 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 20:24:20.829321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.828776 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" podUID="71f8a8a3-fd48-43d2-bf61-87ed31128081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 20:24:20.915297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.906426 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:24:20.916201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:20.916155 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" podUID="87528081-027a-4246-8cf4-70dbea59d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:24:21.171299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:21.157062 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f"] Nov 02 20:24:21.495302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:21.494447 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 20:24:21.495302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:21.494519 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 20:24:21.834302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:21.831632 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Nov 02 20:24:21.834302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:21.831718 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" podUID="71f8a8a3-fd48-43d2-bf61-87ed31128081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Nov 02 20:24:21.872146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:21.872018 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:24:21.872146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:21.872079 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" podUID="87528081-027a-4246-8cf4-70dbea59d933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 20:24:22.013472 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:22.013417 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" event={"ID":"cf408fe6-a44d-479c-9aa6-a9ce068124c5","Type":"ContainerStarted","Data":"d589fca5e4129dffa954bc5aab55267d9d38a667952a23a17a7d79926da84230"} Nov 02 20:24:22.492976 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:22.492943 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 20:24:22.493259 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:22.493217 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 20:24:22.851777 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:22.851733 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:22.852341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:22.852191 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:22.853229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:22.853199 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:24:22.872408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:22.872382 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:22.872673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:22.872651 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:22.873347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:22.873328 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:24:23.022557 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:23.022523 5399 generic.go:356] "Generic (PLEG): container finished" podID="cf408fe6-a44d-479c-9aa6-a9ce068124c5" containerID="5efa2c892ed751ae3a6612fff58adc525c229ce6bb84869affdf248a290384a0" exitCode=0 Nov 02 20:24:23.022751 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:23.022609 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" event={"ID":"cf408fe6-a44d-479c-9aa6-a9ce068124c5","Type":"ContainerDied","Data":"5efa2c892ed751ae3a6612fff58adc525c229ce6bb84869affdf248a290384a0"} Nov 02 20:24:23.491380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:23.491340 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 20:24:23.491573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:23.491398 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 20:24:24.267808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.267782 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:24.361712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.361669 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-openshift-ca\") pod \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " Nov 02 20:24:24.361923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.361890 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf408fe6-a44d-479c-9aa6-a9ce068124c5-kube-api-access-wnhj9\" (UniqueName: \"kubernetes.io/projected/cf408fe6-a44d-479c-9aa6-a9ce068124c5-kube-api-access-wnhj9\") pod \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " Nov 02 20:24:24.362038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.362012 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-openshift-ca" pod "cf408fe6-a44d-479c-9aa6-a9ce068124c5" (UID: "cf408fe6-a44d-479c-9aa6-a9ce068124c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:24.362102 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.362078 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-xks-ca\") pod \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\" (UID: \"cf408fe6-a44d-479c-9aa6-a9ce068124c5\") " Nov 02 20:24:24.362691 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.362669 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:24.363971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.363951 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf408fe6-a44d-479c-9aa6-a9ce068124c5-kube-api-access-wnhj9" pod "cf408fe6-a44d-479c-9aa6-a9ce068124c5" (UID: "cf408fe6-a44d-479c-9aa6-a9ce068124c5"). InnerVolumeSpecName "kube-api-access-wnhj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:24.364134 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.364105 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-xks-ca" pod "cf408fe6-a44d-479c-9aa6-a9ce068124c5" (UID: "cf408fe6-a44d-479c-9aa6-a9ce068124c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:24:24.464521 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.464487 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnhj9\" (UniqueName: \"kubernetes.io/projected/cf408fe6-a44d-479c-9aa6-a9ce068124c5-kube-api-access-wnhj9\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:24.464521 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.464518 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf408fe6-a44d-479c-9aa6-a9ce068124c5-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:24.491887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.491839 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 20:24:24.492080 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:24.491900 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 20:24:25.061967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:25.061928 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" Nov 02 20:24:25.061967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:25.061945 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f" event={"ID":"cf408fe6-a44d-479c-9aa6-a9ce068124c5","Type":"ContainerDied","Data":"d589fca5e4129dffa954bc5aab55267d9d38a667952a23a17a7d79926da84230"} Nov 02 20:24:25.062230 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:25.061991 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d589fca5e4129dffa954bc5aab55267d9d38a667952a23a17a7d79926da84230" Nov 02 20:24:25.491717 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:25.491631 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 20:24:25.491717 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:25.491686 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 20:24:26.501612 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:26.501578 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:26.503267 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:26.503240 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:26.505735 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:26.505708 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:24:36.257309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:36.257260 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs"] Nov 02 20:24:36.257832 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:36.257475 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" containerID="cri-o://f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426" gracePeriod=30 Nov 02 20:24:40.325072 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:40.325032 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:24:40.325474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:40.325102 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:24:41.538725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.538662 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:24:41.620673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.620634 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-socket\") pod \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " Nov 02 20:24:41.620866 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.620753 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-kube-api-access-j49bj\" (UniqueName: \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-kube-api-access-j49bj\") pod \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " Nov 02 20:24:41.620866 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.620792 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-credential-socket\") pod \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " Nov 02 20:24:41.620982 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.620890 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-token\" (UniqueName: \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-token\") pod \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " Nov 02 20:24:41.620982 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.620926 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-podinfo\") pod \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " Nov 02 20:24:41.620982 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.620934 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-socket" pod "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" (UID: "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:41.621141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.620998 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-envoy\") pod \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " Nov 02 20:24:41.621141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.621043 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-data\") pod \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " Nov 02 20:24:41.621141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.621099 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-credential-socket" pod "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" (UID: "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:41.621141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.621122 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istiod-ca-cert\") pod \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " Nov 02 20:24:41.621402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.621384 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-certs\") pod \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\" (UID: \"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb\") " Nov 02 20:24:41.621402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.621395 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-data" pod "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" (UID: "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:41.621515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.621500 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istiod-ca-cert" pod "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" (UID: "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:41.621654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.621620 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-certs" pod "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" (UID: "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:41.622349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.622328 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:41.622462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.622354 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:41.622462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.622368 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:41.622462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.622381 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:41.622462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.622394 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:41.623718 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.623686 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-podinfo" pod "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" (UID: "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:24:41.623817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.623743 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-envoy" pod "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" (UID: "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:41.623817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.623758 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-kube-api-access-j49bj" pod "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" (UID: "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb"). InnerVolumeSpecName "kube-api-access-j49bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:41.623817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.623781 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-token" pod "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" (UID: "c1ab73d4-d368-4b9f-aff9-29f2711cb7fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:41.724191 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.724159 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49bj\" (UniqueName: \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-kube-api-access-j49bj\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:41.724191 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.724191 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:41.724403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.724201 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:41.724403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:41.724210 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:42.194690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:42.194648 5399 generic.go:356] "Generic (PLEG): container finished" podID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerID="f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426" exitCode=0 Nov 02 20:24:42.195114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:42.195092 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" event={"ID":"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb","Type":"ContainerDied","Data":"f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426"} Nov 02 20:24:42.195350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:42.195330 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" event={"ID":"c1ab73d4-d368-4b9f-aff9-29f2711cb7fb","Type":"ContainerDied","Data":"e1463102f0bd14c09c73af3892d75cc335cf8138c8b5ba3f37492610f1be8251"} Nov 02 20:24:42.195492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:42.195394 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs" Nov 02 20:24:42.201145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:42.195414 5399 scope.go:122] "RemoveContainer" containerID="f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426" Nov 02 20:24:42.214319 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:42.214296 5399 scope.go:122] "RemoveContainer" containerID="f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426" Nov 02 20:24:42.214686 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:24:42.214661 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426\": container with ID starting with f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426 not found: ID does not exist" containerID="f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426" Nov 02 20:24:42.214741 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:42.214691 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426"} err="failed to get container status \"f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426\": rpc error: code = NotFound desc = could not find container \"f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426\": container with ID starting with f95ffc8b9b0c6712bec63078b51c77fd595b61c6ae01e9701be1c1b0e5d40426 not found: ID does not exist" Nov 02 20:24:42.357062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:42.357022 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs"] Nov 02 20:24:42.360771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:42.360739 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-srwjs"] Nov 02 20:24:43.513643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:43.513605 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" path="/var/lib/kubelet/pods/c1ab73d4-d368-4b9f-aff9-29f2711cb7fb/volumes" Nov 02 20:24:45.758600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.758562 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm"] Nov 02 20:24:45.759220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.759205 5399 cpu_manager.go:414] "containerMap: removing container" podUID="cf408fe6-a44d-479c-9aa6-a9ce068124c5" containerName="revoke-keys" Nov 02 20:24:45.759269 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.759221 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf408fe6-a44d-479c-9aa6-a9ce068124c5" containerName="revoke-keys" Nov 02 20:24:45.759343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.759297 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" Nov 02 20:24:45.759343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.759305 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" Nov 02 20:24:45.759464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.759453 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf408fe6-a44d-479c-9aa6-a9ce068124c5" containerName="revoke-keys" Nov 02 20:24:45.759500 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.759473 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ab73d4-d368-4b9f-aff9-29f2711cb7fb" containerName="istio-proxy" Nov 02 20:24:45.764791 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.764772 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.884691 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.884655 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.884691 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.884699 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.884960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.884731 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6x2\" (UniqueName: \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-kube-api-access-md6x2\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.885047 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.884979 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.885047 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.885030 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.885164 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.885116 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.885224 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.885168 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de626de0-9f65-47f7-a705-f31a45b24f86-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.885224 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.885191 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.885360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.885260 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de626de0-9f65-47f7-a705-f31a45b24f86-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.956770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.956732 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm"] Nov 02 20:24:45.986735 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.986699 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.986735 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.986737 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.986995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.986805 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.986995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.986840 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de626de0-9f65-47f7-a705-f31a45b24f86-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.986995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.986855 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.986995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.986904 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de626de0-9f65-47f7-a705-f31a45b24f86-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.986995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.986989 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.987314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.987024 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.987314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.987156 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-certs\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.987314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.987223 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6x2\" (UniqueName: \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-kube-api-access-md6x2\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.987493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.987358 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.987493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.987406 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-data\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.987605 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.987508 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-credential-socket\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.987720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.987694 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de626de0-9f65-47f7-a705-f31a45b24f86-istiod-ca-cert\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.989330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.989313 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-envoy\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:45.989402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:45.989376 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de626de0-9f65-47f7-a705-f31a45b24f86-istio-podinfo\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:46.040010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:46.039926 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-istio-token\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:46.040177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:46.040138 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6x2\" (UniqueName: \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-kube-api-access-md6x2\") pod \"e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:46.080101 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:46.080062 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:46.507118 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:46.507091 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm"] Nov 02 20:24:46.509635 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:24:46.509606 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde626de0_9f65_47f7_a705_f31a45b24f86.slice/crio-8d06cccef49f0b7dbd08e53c8ba71148320b8cb505d3865743197a229d62d8ba WatchSource:0}: Error finding container 8d06cccef49f0b7dbd08e53c8ba71148320b8cb505d3865743197a229d62d8ba: Status 404 returned error can't find the container with id 8d06cccef49f0b7dbd08e53c8ba71148320b8cb505d3865743197a229d62d8ba Nov 02 20:24:46.512607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:46.512573 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:46.512683 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:46.512646 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:46.512683 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:46.512674 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:47.259905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:47.259864 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" event={"ID":"de626de0-9f65-47f7-a705-f31a45b24f86","Type":"ContainerStarted","Data":"01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59"} Nov 02 20:24:47.260446 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:47.259932 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" event={"ID":"de626de0-9f65-47f7-a705-f31a45b24f86","Type":"ContainerStarted","Data":"8d06cccef49f0b7dbd08e53c8ba71148320b8cb505d3865743197a229d62d8ba"} Nov 02 20:24:47.464954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:47.464876 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" podStartSLOduration=2.464856061 podStartE2EDuration="2.464856061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:47.416750368 +0000 UTC m=+1930.027402499" watchObservedRunningTime="2026-11-02 20:24:47.464856061 +0000 UTC m=+1930.075508156" Nov 02 20:24:48.080438 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.080392 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:48.081957 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.081932 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 20:24:48.082017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.081996 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" podUID="de626de0-9f65-47f7-a705-f31a45b24f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 20:24:48.258217 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:24:48.258165 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:24:48.362606 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.362490 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d"] Nov 02 20:24:48.368062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.368033 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.409626 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.409592 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d"] Nov 02 20:24:48.422778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.422746 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.423047 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.423008 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.423144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.423058 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.423144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.423104 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.423144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.423132 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhnk\" (UniqueName: \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-kube-api-access-mfhnk\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.423334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.423314 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.423502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.423467 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.423666 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.423639 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.423775 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.423751 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b27d65e-440e-41df-b238-c9cbdd902ed2-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.524649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.524612 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.524856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.524662 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.524856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.524710 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.524856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.524734 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhnk\" (UniqueName: \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-kube-api-access-mfhnk\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.524856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.524822 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.525076 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.525051 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.525136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.525098 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.525136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.525118 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-certs\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.525250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.525195 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.525334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.525059 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-credential-socket\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.525394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.525346 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b27d65e-440e-41df-b238-c9cbdd902ed2-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.525461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.525435 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.525816 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.525787 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-data\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.526000 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.525973 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b27d65e-440e-41df-b238-c9cbdd902ed2-istiod-ca-cert\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.556783 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.556738 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-envoy\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.557494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.557469 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-podinfo\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.577677 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.577649 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-token\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.577801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.577773 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhnk\" (UniqueName: \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-kube-api-access-mfhnk\") pod \"e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:48.684930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:48.684840 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:49.081008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:49.080975 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 20:24:49.081187 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:49.081027 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" podUID="de626de0-9f65-47f7-a705-f31a45b24f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 20:24:49.151323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:49.150787 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d"] Nov 02 20:24:49.163590 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:24:49.163558 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b27d65e_440e_41df_b238_c9cbdd902ed2.slice/crio-89822f500178474c403aea29d5c174fffe0e31b5cdf26f35da3e09c4001d9e0d WatchSource:0}: Error finding container 89822f500178474c403aea29d5c174fffe0e31b5cdf26f35da3e09c4001d9e0d: Status 404 returned error can't find the container with id 89822f500178474c403aea29d5c174fffe0e31b5cdf26f35da3e09c4001d9e0d Nov 02 20:24:49.167962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:49.167920 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:49.168073 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:49.168036 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:49.168115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:49.168079 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:49.277978 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:49.277939 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" event={"ID":"6b27d65e-440e-41df-b238-c9cbdd902ed2","Type":"ContainerStarted","Data":"da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914"} Nov 02 20:24:49.278153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:49.278005 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" event={"ID":"6b27d65e-440e-41df-b238-c9cbdd902ed2","Type":"ContainerStarted","Data":"89822f500178474c403aea29d5c174fffe0e31b5cdf26f35da3e09c4001d9e0d"} Nov 02 20:24:49.418739 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:49.418612 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" podStartSLOduration=1.418591944 podStartE2EDuration="1.418591944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:49.409467269 +0000 UTC m=+1932.020119361" watchObservedRunningTime="2026-11-02 20:24:49.418591944 +0000 UTC m=+1932.029244036" Nov 02 20:24:49.685678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:49.685602 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:50.080828 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:50.080779 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 20:24:50.081017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:50.080851 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" podUID="de626de0-9f65-47f7-a705-f31a45b24f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 20:24:50.686769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:50.686733 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:24:50.687177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:50.686789 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" podUID="6b27d65e-440e-41df-b238-c9cbdd902ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 20:24:51.081468 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:51.081437 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 20:24:51.081725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:51.081699 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" podUID="de626de0-9f65-47f7-a705-f31a45b24f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 20:24:51.685643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:51.685602 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:24:51.685831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:51.685652 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" podUID="6b27d65e-440e-41df-b238-c9cbdd902ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 20:24:52.082216 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:52.081606 5399 patch_prober.go:29] interesting pod/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 20:24:52.082216 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:52.081681 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" podUID="de626de0-9f65-47f7-a705-f31a45b24f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 20:24:52.686306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:52.685990 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:24:52.686306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:52.686051 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" podUID="6b27d65e-440e-41df-b238-c9cbdd902ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 20:24:53.207317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:53.207269 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:53.208703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:53.208668 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:53.209210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:53.209189 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:24:53.686249 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:53.686206 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:24:53.686466 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:53.686347 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" podUID="6b27d65e-440e-41df-b238-c9cbdd902ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 20:24:54.685415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:54.685381 5399 patch_prober.go:29] interesting pod/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 20:24:54.685899 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:54.685439 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" podUID="6b27d65e-440e-41df-b238-c9cbdd902ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 20:24:55.689828 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:55.689798 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:55.690202 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:55.690120 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:24:55.690986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:24:55.690964 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:25:02.232975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.232934 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh"] Nov 02 20:25:02.237902 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.237874 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:25:02.288097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.288047 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-81e58f\"" Nov 02 20:25:02.334603 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.334572 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-81e58f-dockercfg-lts5q\"" Nov 02 20:25:02.342421 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.342391 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh"] Nov 02 20:25:02.372100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.372040 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4prd\" (UniqueName: \"kubernetes.io/projected/52bd4f76-0054-4157-8ab2-f801a14b06ea-kube-api-access-d4prd\") pod \"payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh\" (UID: \"52bd4f76-0054-4157-8ab2-f801a14b06ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:25:02.372335 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.372196 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bd4f76-0054-4157-8ab2-f801a14b06ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh\" (UID: \"52bd4f76-0054-4157-8ab2-f801a14b06ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:25:02.484391 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.475690 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4prd\" (UniqueName: \"kubernetes.io/projected/52bd4f76-0054-4157-8ab2-f801a14b06ea-kube-api-access-d4prd\") pod \"payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh\" (UID: \"52bd4f76-0054-4157-8ab2-f801a14b06ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:25:02.485793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.485447 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bd4f76-0054-4157-8ab2-f801a14b06ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh\" (UID: \"52bd4f76-0054-4157-8ab2-f801a14b06ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:25:02.498459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.498138 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bd4f76-0054-4157-8ab2-f801a14b06ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh\" (UID: \"52bd4f76-0054-4157-8ab2-f801a14b06ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:25:02.539700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.538554 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4prd\" (UniqueName: \"kubernetes.io/projected/52bd4f76-0054-4157-8ab2-f801a14b06ea-kube-api-access-d4prd\") pod \"payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh\" (UID: \"52bd4f76-0054-4157-8ab2-f801a14b06ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:25:02.555020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.554987 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:25:02.990333 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:25:02.990296 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bd4f76_0054_4157_8ab2_f801a14b06ea.slice/crio-85496fb70c09064b7c43e318e101b3d39f66cca1f3676ddc60980d7d62c8e385 WatchSource:0}: Error finding container 85496fb70c09064b7c43e318e101b3d39f66cca1f3676ddc60980d7d62c8e385: Status 404 returned error can't find the container with id 85496fb70c09064b7c43e318e101b3d39f66cca1f3676ddc60980d7d62c8e385 Nov 02 20:25:02.990642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:02.990620 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh"] Nov 02 20:25:03.445238 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:03.445196 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" event={"ID":"52bd4f76-0054-4157-8ab2-f801a14b06ea","Type":"ContainerStarted","Data":"eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0"} Nov 02 20:25:03.445658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:03.445263 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" event={"ID":"52bd4f76-0054-4157-8ab2-f801a14b06ea","Type":"ContainerStarted","Data":"85496fb70c09064b7c43e318e101b3d39f66cca1f3676ddc60980d7d62c8e385"} Nov 02 20:25:03.445658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:03.445303 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:25:03.551794 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:03.551729 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" podStartSLOduration=1.551709596 podStartE2EDuration="1.551709596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:03.550575422 +0000 UTC m=+1946.161227513" watchObservedRunningTime="2026-11-02 20:25:03.551709596 +0000 UTC m=+1946.162361690" Nov 02 20:25:06.516668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:06.516624 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:25:06.941314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:06.941264 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g"] Nov 02 20:25:06.946189 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:06.946166 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:06.998162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:06.998131 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8c5e8e-openshift-default-dockercfg-jgkzj\"" Nov 02 20:25:07.012133 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.012097 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-socket\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.012541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.012515 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqll\" (UniqueName: \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-kube-api-access-fjqll\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.012735 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.012718 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-data\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.012918 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.012902 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-podinfo\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.013140 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.013124 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-credential-socket\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.013300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.013266 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-certs\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.013450 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.013435 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-token\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.013601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.013584 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a960d8-7c24-462a-9a0d-e1205a4ee579-istiod-ca-cert\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.013740 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.013725 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-envoy\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.106937 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.086688 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g"] Nov 02 20:25:07.120771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.120729 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-socket\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.121021 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.121001 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqll\" (UniqueName: \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-kube-api-access-fjqll\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.121170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.121153 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-data\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.121332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.121315 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-podinfo\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.121540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.121524 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-credential-socket\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.121647 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.121634 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-certs\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.121745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.121732 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-token\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.121858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.121843 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a960d8-7c24-462a-9a0d-e1205a4ee579-istiod-ca-cert\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.121961 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.121947 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-envoy\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.126232 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.126209 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-certs\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.126678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.126656 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-credential-socket\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.127368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.127345 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-socket\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.131113 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.131074 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a960d8-7c24-462a-9a0d-e1205a4ee579-istiod-ca-cert\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.131555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.131508 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-data\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.138123 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.138099 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-envoy\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.171330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.168735 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-podinfo\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.185097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.185034 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqll\" (UniqueName: \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-kube-api-access-fjqll\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.189039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.188943 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-token\") pod \"e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.261502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.261339 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:07.710988 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:25:07.710953 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a960d8_7c24_462a_9a0d_e1205a4ee579.slice/crio-b67bbc4bec1c6798316b2624039de36d1942f04e5ee8b42e1dc2e94585a66466 WatchSource:0}: Error finding container b67bbc4bec1c6798316b2624039de36d1942f04e5ee8b42e1dc2e94585a66466: Status 404 returned error can't find the container with id b67bbc4bec1c6798316b2624039de36d1942f04e5ee8b42e1dc2e94585a66466 Nov 02 20:25:07.711836 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.711808 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g"] Nov 02 20:25:07.714075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.714043 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:25:07.714159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.714107 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:25:07.714159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:07.714133 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:25:08.490693 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:08.490654 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" event={"ID":"81a960d8-7c24-462a-9a0d-e1205a4ee579","Type":"ContainerStarted","Data":"ddfa6626bc66e0a91a179e1cf437117f535630e3439013fac48a44691d092886"} Nov 02 20:25:08.490902 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:08.490719 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" event={"ID":"81a960d8-7c24-462a-9a0d-e1205a4ee579","Type":"ContainerStarted","Data":"b67bbc4bec1c6798316b2624039de36d1942f04e5ee8b42e1dc2e94585a66466"} Nov 02 20:25:08.606827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:08.606772 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" podStartSLOduration=2.606755696 podStartE2EDuration="2.606755696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:08.602178485 +0000 UTC m=+1951.212830577" watchObservedRunningTime="2026-11-02 20:25:08.606755696 +0000 UTC m=+1951.217407790" Nov 02 20:25:09.261721 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.261683 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:09.265210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.265181 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j"] Nov 02 20:25:09.269592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.269570 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:09.269708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.269637 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.320856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.320090 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-eb060a-openshift-default-dockercfg-wzwf8\"" Nov 02 20:25:09.352631 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.352591 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j"] Nov 02 20:25:09.365574 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.365542 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-socket\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.365767 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.365694 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-certs\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.365832 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.365767 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-credential-socket\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.365832 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.365803 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-podinfo\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.365938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.365847 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istiod-ca-cert\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.365938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.365874 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-envoy\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.365938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.365909 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-data\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.366084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.365984 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-token\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.366084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.366035 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4c6\" (UniqueName: \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-kube-api-access-qj4c6\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.467600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.467567 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4c6\" (UniqueName: \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-kube-api-access-qj4c6\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.467818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.467634 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-socket\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.467818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.467780 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-certs\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.467949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.467837 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-credential-socket\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.467949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.467867 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-podinfo\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.467949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.467903 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istiod-ca-cert\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.468148 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.467950 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-envoy\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.468148 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.468002 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-data\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.468148 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.468095 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-socket\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.468148 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.468110 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-certs\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.468384 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.468163 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-token\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.468384 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.468228 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-credential-socket\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.468472 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.468445 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-data\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.469062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.469037 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istiod-ca-cert\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.470372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.470345 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-envoy\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.470764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.470742 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-podinfo\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.510592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.510553 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:09.510773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.510645 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:25:09.521061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.520980 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-token\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.521061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.521048 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4c6\" (UniqueName: \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-kube-api-access-qj4c6\") pod \"e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:09.584543 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:09.584510 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:10.021799 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:25:10.021763 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db637dd_249a_4b6f_9a3c_5b00cd9bc7f5.slice/crio-0588e20adfae2c58c567ebff5e8096e0a3a18751cee30beab17b75f38562dfe6 WatchSource:0}: Error finding container 0588e20adfae2c58c567ebff5e8096e0a3a18751cee30beab17b75f38562dfe6: Status 404 returned error can't find the container with id 0588e20adfae2c58c567ebff5e8096e0a3a18751cee30beab17b75f38562dfe6 Nov 02 20:25:10.022080 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:10.022058 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j"] Nov 02 20:25:10.024601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:10.024574 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:25:10.024670 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:10.024634 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:25:10.024670 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:10.024660 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:25:10.512491 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:10.512448 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" event={"ID":"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5","Type":"ContainerStarted","Data":"01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2"} Nov 02 20:25:10.512994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:10.512511 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" event={"ID":"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5","Type":"ContainerStarted","Data":"0588e20adfae2c58c567ebff5e8096e0a3a18751cee30beab17b75f38562dfe6"} Nov 02 20:25:10.585363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:10.585323 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:10.621597 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:10.621511 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" podStartSLOduration=1.621495884 podStartE2EDuration="1.621495884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:10.619535113 +0000 UTC m=+1953.230187237" watchObservedRunningTime="2026-11-02 20:25:10.621495884 +0000 UTC m=+1953.232147975" Nov 02 20:25:11.591476 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:11.591443 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:12.527751 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:12.527722 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:12.528656 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:12.528638 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:25:14.456069 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:14.456041 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:25:15.783907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:15.783865 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl"] Nov 02 20:25:15.789850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:15.789827 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:25:15.840530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:15.840488 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl"] Nov 02 20:25:15.881677 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:15.881526 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-429b45\"" Nov 02 20:25:15.881943 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:15.881582 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-429b45-dockercfg-qz9k2\"" Nov 02 20:25:15.887231 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:15.887176 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d7a94c-d41d-4122-8274-aec68f197f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl\" (UID: \"82d7a94c-d41d-4122-8274-aec68f197f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:25:15.888098 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:15.888046 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj54k\" (UniqueName: \"kubernetes.io/projected/82d7a94c-d41d-4122-8274-aec68f197f9f-kube-api-access-kj54k\") pod \"payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl\" (UID: \"82d7a94c-d41d-4122-8274-aec68f197f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:25:15.994305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:15.990727 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj54k\" (UniqueName: \"kubernetes.io/projected/82d7a94c-d41d-4122-8274-aec68f197f9f-kube-api-access-kj54k\") pod \"payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl\" (UID: \"82d7a94c-d41d-4122-8274-aec68f197f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:25:15.994305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:15.990843 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d7a94c-d41d-4122-8274-aec68f197f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl\" (UID: \"82d7a94c-d41d-4122-8274-aec68f197f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:25:15.998886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:15.998850 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d7a94c-d41d-4122-8274-aec68f197f9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl\" (UID: \"82d7a94c-d41d-4122-8274-aec68f197f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:25:16.046297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:16.046193 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj54k\" (UniqueName: \"kubernetes.io/projected/82d7a94c-d41d-4122-8274-aec68f197f9f-kube-api-access-kj54k\") pod \"payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl\" (UID: \"82d7a94c-d41d-4122-8274-aec68f197f9f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:25:16.108669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:16.108636 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:25:16.484876 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:16.484843 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl"] Nov 02 20:25:16.486077 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:25:16.486045 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d7a94c_d41d_4122_8274_aec68f197f9f.slice/crio-9389c4d4b2f552d75867c4f089ea78accc3d797ac31f74fb43797e43232f8eb3 WatchSource:0}: Error finding container 9389c4d4b2f552d75867c4f089ea78accc3d797ac31f74fb43797e43232f8eb3: Status 404 returned error can't find the container with id 9389c4d4b2f552d75867c4f089ea78accc3d797ac31f74fb43797e43232f8eb3 Nov 02 20:25:16.559338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:16.559306 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" event={"ID":"82d7a94c-d41d-4122-8274-aec68f197f9f","Type":"ContainerStarted","Data":"9389c4d4b2f552d75867c4f089ea78accc3d797ac31f74fb43797e43232f8eb3"} Nov 02 20:25:17.650771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:17.649575 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" event={"ID":"82d7a94c-d41d-4122-8274-aec68f197f9f","Type":"ContainerStarted","Data":"c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737"} Nov 02 20:25:17.650771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:17.650161 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:25:17.757798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:17.757740 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" podStartSLOduration=2.757723346 podStartE2EDuration="2.757723346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:17.755580339 +0000 UTC m=+1960.366232433" watchObservedRunningTime="2026-11-02 20:25:17.757723346 +0000 UTC m=+1960.368375436" Nov 02 20:25:18.260300 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:25:18.259832 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:25:28.690068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:28.690036 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:25:28.993684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:28.993593 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd"] Nov 02 20:25:28.999438 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:28.999394 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:25:29.052298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.051575 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-eb060a\"" Nov 02 20:25:29.052298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.051764 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-eb060a-dockercfg-wfzkd\"" Nov 02 20:25:29.064337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.053239 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ckn\" (UniqueName: \"kubernetes.io/projected/3313660b-c048-4c33-8524-30f760d0b1e0-kube-api-access-45ckn\") pod \"payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd\" (UID: \"3313660b-c048-4c33-8524-30f760d0b1e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:25:29.064337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.053483 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3313660b-c048-4c33-8524-30f760d0b1e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd\" (UID: \"3313660b-c048-4c33-8524-30f760d0b1e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:25:29.082826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.073778 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd"] Nov 02 20:25:29.194298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.191713 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45ckn\" (UniqueName: \"kubernetes.io/projected/3313660b-c048-4c33-8524-30f760d0b1e0-kube-api-access-45ckn\") pod \"payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd\" (UID: \"3313660b-c048-4c33-8524-30f760d0b1e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:25:29.194298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.191934 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3313660b-c048-4c33-8524-30f760d0b1e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd\" (UID: \"3313660b-c048-4c33-8524-30f760d0b1e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:25:29.194298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.193021 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3313660b-c048-4c33-8524-30f760d0b1e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd\" (UID: \"3313660b-c048-4c33-8524-30f760d0b1e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:25:29.225298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.224911 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f"] Nov 02 20:25:29.242853 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.242668 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:25:29.317402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.305677 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec25bbd4-d01c-48f2-840a-81e159db4939-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f\" (UID: \"ec25bbd4-d01c-48f2-840a-81e159db4939\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:25:29.317402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.308585 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tst4m\" (UniqueName: \"kubernetes.io/projected/ec25bbd4-d01c-48f2-840a-81e159db4939-kube-api-access-tst4m\") pod \"payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f\" (UID: \"ec25bbd4-d01c-48f2-840a-81e159db4939\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:25:29.320474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.320402 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f"] Nov 02 20:25:29.321508 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.321485 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ckn\" (UniqueName: \"kubernetes.io/projected/3313660b-c048-4c33-8524-30f760d0b1e0-kube-api-access-45ckn\") pod \"payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd\" (UID: \"3313660b-c048-4c33-8524-30f760d0b1e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:25:29.337539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.335869 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:25:29.410770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.410111 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec25bbd4-d01c-48f2-840a-81e159db4939-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f\" (UID: \"ec25bbd4-d01c-48f2-840a-81e159db4939\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:25:29.410770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.410391 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tst4m\" (UniqueName: \"kubernetes.io/projected/ec25bbd4-d01c-48f2-840a-81e159db4939-kube-api-access-tst4m\") pod \"payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f\" (UID: \"ec25bbd4-d01c-48f2-840a-81e159db4939\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:25:29.411128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.411102 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec25bbd4-d01c-48f2-840a-81e159db4939-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f\" (UID: \"ec25bbd4-d01c-48f2-840a-81e159db4939\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:25:29.466226 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.466151 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tst4m\" (UniqueName: \"kubernetes.io/projected/ec25bbd4-d01c-48f2-840a-81e159db4939-kube-api-access-tst4m\") pod \"payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f\" (UID: \"ec25bbd4-d01c-48f2-840a-81e159db4939\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:25:29.642372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.642295 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:25:29.731524 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.731430 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd"] Nov 02 20:25:29.735361 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:25:29.735334 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3313660b_c048_4c33_8524_30f760d0b1e0.slice/crio-70d6c106b18fe23a53fdfc8744d44209d2c6c25e63a392a8300d58e54980d6f1 WatchSource:0}: Error finding container 70d6c106b18fe23a53fdfc8744d44209d2c6c25e63a392a8300d58e54980d6f1: Status 404 returned error can't find the container with id 70d6c106b18fe23a53fdfc8744d44209d2c6c25e63a392a8300d58e54980d6f1 Nov 02 20:25:29.785139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:29.785103 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" event={"ID":"3313660b-c048-4c33-8524-30f760d0b1e0","Type":"ContainerStarted","Data":"70d6c106b18fe23a53fdfc8744d44209d2c6c25e63a392a8300d58e54980d6f1"} Nov 02 20:25:30.017189 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:25:30.017158 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec25bbd4_d01c_48f2_840a_81e159db4939.slice/crio-5d8026217e6bd478c65741c0c7e6d1e03784df031b22b455e3034da3fe89e95a WatchSource:0}: Error finding container 5d8026217e6bd478c65741c0c7e6d1e03784df031b22b455e3034da3fe89e95a: Status 404 returned error can't find the container with id 5d8026217e6bd478c65741c0c7e6d1e03784df031b22b455e3034da3fe89e95a Nov 02 20:25:30.017189 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:30.017163 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f"] Nov 02 20:25:30.794609 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:30.794571 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" event={"ID":"3313660b-c048-4c33-8524-30f760d0b1e0","Type":"ContainerStarted","Data":"71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1"} Nov 02 20:25:30.795063 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:30.794639 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:25:30.796358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:30.796325 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" event={"ID":"ec25bbd4-d01c-48f2-840a-81e159db4939","Type":"ContainerStarted","Data":"c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a"} Nov 02 20:25:30.796486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:30.796373 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" event={"ID":"ec25bbd4-d01c-48f2-840a-81e159db4939","Type":"ContainerStarted","Data":"5d8026217e6bd478c65741c0c7e6d1e03784df031b22b455e3034da3fe89e95a"} Nov 02 20:25:30.796486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:30.796401 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:25:30.902444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:30.902392 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" podStartSLOduration=2.902376595 podStartE2EDuration="2.902376595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:30.901213077 +0000 UTC m=+1973.511865169" watchObservedRunningTime="2026-11-02 20:25:30.902376595 +0000 UTC m=+1973.513028685" Nov 02 20:25:31.027373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:31.023397 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" podStartSLOduration=2.023373774 podStartE2EDuration="2.023373774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:31.009073866 +0000 UTC m=+1973.619725958" watchObservedRunningTime="2026-11-02 20:25:31.023373774 +0000 UTC m=+1973.634025867" Nov 02 20:25:38.302030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:38.301985 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7"] Nov 02 20:25:38.306669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:38.306596 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" podUID="43cff83a-d81e-49bb-b7e1-956e820bc615" containerName="istio-proxy" containerID="cri-o://c6c23cb8694700a9c19628b1bd83f8dc928eb2cb23b1321bc22eedaa2c3d7076" gracePeriod=30 Nov 02 20:25:41.807405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:41.807366 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:25:41.807819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:41.807445 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:25:42.371931 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.371895 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht"] Nov 02 20:25:42.372172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.372149 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" podUID="dc9560d7-7263-49b9-99e4-012124e9a461" containerName="maas-api" containerID="cri-o://aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70" gracePeriod=30 Nov 02 20:25:42.682350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.682263 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:25:42.742808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.742744 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9560d7-7263-49b9-99e4-012124e9a461-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc9560d7-7263-49b9-99e4-012124e9a461-openshift-service-ca\") pod \"dc9560d7-7263-49b9-99e4-012124e9a461\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " Nov 02 20:25:42.743007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.742982 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9560d7-7263-49b9-99e4-012124e9a461-kube-api-access-7bqk7\" (UniqueName: \"kubernetes.io/projected/dc9560d7-7263-49b9-99e4-012124e9a461-kube-api-access-7bqk7\") pod \"dc9560d7-7263-49b9-99e4-012124e9a461\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " Nov 02 20:25:42.743221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.743191 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-metrics-certs\") pod \"dc9560d7-7263-49b9-99e4-012124e9a461\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " Nov 02 20:25:42.743372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.743226 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9560d7-7263-49b9-99e4-012124e9a461-openshift-service-ca" pod "dc9560d7-7263-49b9-99e4-012124e9a461" (UID: "dc9560d7-7263-49b9-99e4-012124e9a461"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:42.743372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.743254 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls\") pod \"dc9560d7-7263-49b9-99e4-012124e9a461\" (UID: \"dc9560d7-7263-49b9-99e4-012124e9a461\") " Nov 02 20:25:42.743949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.743926 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc9560d7-7263-49b9-99e4-012124e9a461-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:42.745642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.745597 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls" pod "dc9560d7-7263-49b9-99e4-012124e9a461" (UID: "dc9560d7-7263-49b9-99e4-012124e9a461"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:25:42.746232 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.746206 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9560d7-7263-49b9-99e4-012124e9a461-kube-api-access-7bqk7" pod "dc9560d7-7263-49b9-99e4-012124e9a461" (UID: "dc9560d7-7263-49b9-99e4-012124e9a461"). InnerVolumeSpecName "kube-api-access-7bqk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:42.746593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.746564 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-metrics-certs" pod "dc9560d7-7263-49b9-99e4-012124e9a461" (UID: "dc9560d7-7263-49b9-99e4-012124e9a461"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:25:42.862549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.862514 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:42.863166 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.863099 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc9560d7-7263-49b9-99e4-012124e9a461-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:42.863366 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.863351 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bqk7\" (UniqueName: \"kubernetes.io/projected/dc9560d7-7263-49b9-99e4-012124e9a461-kube-api-access-7bqk7\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:42.888443 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.888261 5399 generic.go:356] "Generic (PLEG): container finished" podID="dc9560d7-7263-49b9-99e4-012124e9a461" containerID="aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70" exitCode=0 Nov 02 20:25:42.888443 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.888360 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" event={"ID":"dc9560d7-7263-49b9-99e4-012124e9a461","Type":"ContainerDied","Data":"aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70"} Nov 02 20:25:42.888443 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.888377 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" Nov 02 20:25:42.888443 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.888440 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" event={"ID":"dc9560d7-7263-49b9-99e4-012124e9a461","Type":"ContainerDied","Data":"81ae9d65a3909690813f9fbeddf523e9f8d419e9c259d90eeccc5f96d380aa7e"} Nov 02 20:25:42.888800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.888471 5399 scope.go:122] "RemoveContainer" containerID="aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70" Nov 02 20:25:42.905788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.905751 5399 scope.go:122] "RemoveContainer" containerID="aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70" Nov 02 20:25:42.906864 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:25:42.906178 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70\": container with ID starting with aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70 not found: ID does not exist" containerID="aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70" Nov 02 20:25:42.906864 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:42.906224 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70"} err="failed to get container status \"aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70\": rpc error: code = NotFound desc = could not find container \"aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70\": container with ID starting with aade952be5506b367adce08a445414a4261a2770a001b0e968698eebcd64ec70 not found: ID does not exist" Nov 02 20:25:43.044542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.044506 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht"] Nov 02 20:25:43.048804 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.048771 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht"] Nov 02 20:25:43.634868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.634750 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9560d7-7263-49b9-99e4-012124e9a461" path="/var/lib/kubelet/pods/dc9560d7-7263-49b9-99e4-012124e9a461/volumes" Nov 02 20:25:43.660765 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.660717 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4"] Nov 02 20:25:43.661901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.661876 5399 cpu_manager.go:414] "containerMap: removing container" podUID="dc9560d7-7263-49b9-99e4-012124e9a461" containerName="maas-api" Nov 02 20:25:43.662014 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.661904 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9560d7-7263-49b9-99e4-012124e9a461" containerName="maas-api" Nov 02 20:25:43.662106 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.662091 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9560d7-7263-49b9-99e4-012124e9a461" containerName="maas-api" Nov 02 20:25:43.669263 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.669232 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:25:43.718025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.717877 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4"] Nov 02 20:25:43.719072 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.718014 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d5a51f-dockercfg-6lz9b\"" Nov 02 20:25:43.719255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.718207 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d5a51f\"" Nov 02 20:25:43.770821 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.765568 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4\" (UID: \"16fdd199-f388-4ed2-abe5-2c2e5acb6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:25:43.770821 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.765835 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgkf2\" (UniqueName: \"kubernetes.io/projected/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-kube-api-access-hgkf2\") pod \"payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4\" (UID: \"16fdd199-f388-4ed2-abe5-2c2e5acb6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:25:43.914161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.914049 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4\" (UID: \"16fdd199-f388-4ed2-abe5-2c2e5acb6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:25:43.914665 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.914251 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgkf2\" (UniqueName: \"kubernetes.io/projected/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-kube-api-access-hgkf2\") pod \"payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4\" (UID: \"16fdd199-f388-4ed2-abe5-2c2e5acb6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:25:43.915785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.915757 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4\" (UID: \"16fdd199-f388-4ed2-abe5-2c2e5acb6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:25:43.918416 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.918383 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr"] Nov 02 20:25:43.924063 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.924047 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:25:43.936998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.936706 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr"] Nov 02 20:25:43.944956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.944387 5399 generic.go:356] "Generic (PLEG): container finished" podID="43cff83a-d81e-49bb-b7e1-956e820bc615" containerID="c6c23cb8694700a9c19628b1bd83f8dc928eb2cb23b1321bc22eedaa2c3d7076" exitCode=0 Nov 02 20:25:43.944956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.944506 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" event={"ID":"43cff83a-d81e-49bb-b7e1-956e820bc615","Type":"ContainerDied","Data":"c6c23cb8694700a9c19628b1bd83f8dc928eb2cb23b1321bc22eedaa2c3d7076"} Nov 02 20:25:43.972711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.972673 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgkf2\" (UniqueName: \"kubernetes.io/projected/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-kube-api-access-hgkf2\") pod \"payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4\" (UID: \"16fdd199-f388-4ed2-abe5-2c2e5acb6a41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:25:43.998993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:43.998950 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:25:44.118311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.111626 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:25:44.129731 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.124946 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr\" (UID: \"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:25:44.129731 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.125201 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxz44\" (UniqueName: \"kubernetes.io/projected/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-kube-api-access-fxz44\") pod \"payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr\" (UID: \"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:25:44.253869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.253832 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-credential-socket\") pod \"43cff83a-d81e-49bb-b7e1-956e820bc615\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " Nov 02 20:25:44.254061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.253908 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-socket\") pod \"43cff83a-d81e-49bb-b7e1-956e820bc615\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " Nov 02 20:25:44.254061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.253949 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-certs\") pod \"43cff83a-d81e-49bb-b7e1-956e820bc615\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " Nov 02 20:25:44.254061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254006 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-envoy\") pod \"43cff83a-d81e-49bb-b7e1-956e820bc615\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " Nov 02 20:25:44.254061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254062 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-kube-api-access-gm27q\" (UniqueName: \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-kube-api-access-gm27q\") pod \"43cff83a-d81e-49bb-b7e1-956e820bc615\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " Nov 02 20:25:44.254258 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254088 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43cff83a-d81e-49bb-b7e1-956e820bc615-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cff83a-d81e-49bb-b7e1-956e820bc615-istiod-ca-cert\") pod \"43cff83a-d81e-49bb-b7e1-956e820bc615\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " Nov 02 20:25:44.254258 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254216 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-token\" (UniqueName: \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-token\") pod \"43cff83a-d81e-49bb-b7e1-956e820bc615\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " Nov 02 20:25:44.254258 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254230 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-socket" pod "43cff83a-d81e-49bb-b7e1-956e820bc615" (UID: "43cff83a-d81e-49bb-b7e1-956e820bc615"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.254426 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254258 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-podinfo\") pod \"43cff83a-d81e-49bb-b7e1-956e820bc615\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " Nov 02 20:25:44.254426 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254330 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-certs" pod "43cff83a-d81e-49bb-b7e1-956e820bc615" (UID: "43cff83a-d81e-49bb-b7e1-956e820bc615"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.254530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254452 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-data\") pod \"43cff83a-d81e-49bb-b7e1-956e820bc615\" (UID: \"43cff83a-d81e-49bb-b7e1-956e820bc615\") " Nov 02 20:25:44.254690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254664 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cff83a-d81e-49bb-b7e1-956e820bc615-istiod-ca-cert" pod "43cff83a-d81e-49bb-b7e1-956e820bc615" (UID: "43cff83a-d81e-49bb-b7e1-956e820bc615"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:44.254800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254778 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-credential-socket" pod "43cff83a-d81e-49bb-b7e1-956e820bc615" (UID: "43cff83a-d81e-49bb-b7e1-956e820bc615"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.254955 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.254939 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-data" pod "43cff83a-d81e-49bb-b7e1-956e820bc615" (UID: "43cff83a-d81e-49bb-b7e1-956e820bc615"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.255152 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.255126 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxz44\" (UniqueName: \"kubernetes.io/projected/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-kube-api-access-fxz44\") pod \"payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr\" (UID: \"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:25:44.255570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.255544 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr\" (UID: \"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:25:44.255689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.255662 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.255689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.255679 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.255809 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.255698 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.255809 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.255712 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cff83a-d81e-49bb-b7e1-956e820bc615-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.255809 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.255726 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.256538 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.256513 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr\" (UID: \"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:25:44.257118 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.257093 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-kube-api-access-gm27q" pod "43cff83a-d81e-49bb-b7e1-956e820bc615" (UID: "43cff83a-d81e-49bb-b7e1-956e820bc615"). InnerVolumeSpecName "kube-api-access-gm27q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:44.257739 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.257710 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-token" pod "43cff83a-d81e-49bb-b7e1-956e820bc615" (UID: "43cff83a-d81e-49bb-b7e1-956e820bc615"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:44.258577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.258552 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-envoy" pod "43cff83a-d81e-49bb-b7e1-956e820bc615" (UID: "43cff83a-d81e-49bb-b7e1-956e820bc615"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:44.258674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.258617 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-podinfo" pod "43cff83a-d81e-49bb-b7e1-956e820bc615" (UID: "43cff83a-d81e-49bb-b7e1-956e820bc615"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:25:44.308155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.308116 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxz44\" (UniqueName: \"kubernetes.io/projected/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-kube-api-access-fxz44\") pod \"payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr\" (UID: \"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:25:44.357286 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.357240 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.357286 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.357289 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm27q\" (UniqueName: \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-kube-api-access-gm27q\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.357482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.357307 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.357482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.357318 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cff83a-d81e-49bb-b7e1-956e820bc615-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:44.508217 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.498610 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4"] Nov 02 20:25:44.545329 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.545296 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:25:44.954169 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.954127 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" event={"ID":"16fdd199-f388-4ed2-abe5-2c2e5acb6a41","Type":"ContainerStarted","Data":"9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e"} Nov 02 20:25:44.954735 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.954198 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" event={"ID":"16fdd199-f388-4ed2-abe5-2c2e5acb6a41","Type":"ContainerStarted","Data":"c50416f14a9ae0f383abf102baee1a0b3efa1e61e1a915189e90a01d3ba2c4ea"} Nov 02 20:25:44.954735 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.954229 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:25:44.955861 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.955836 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" event={"ID":"43cff83a-d81e-49bb-b7e1-956e820bc615","Type":"ContainerDied","Data":"cdc2e2e865faf52327bc5ae27f7fabdaa59674db964bfb1d470a9c8c77b00850"} Nov 02 20:25:44.955985 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.955862 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7" Nov 02 20:25:44.955985 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.955888 5399 scope.go:122] "RemoveContainer" containerID="c6c23cb8694700a9c19628b1bd83f8dc928eb2cb23b1321bc22eedaa2c3d7076" Nov 02 20:25:44.976363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:44.976334 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr"] Nov 02 20:25:44.976650 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:25:44.976630 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f42b0a8_fac1_425f_9b79_0dba2dca2fa4.slice/crio-566dfa24242e3db32d38516025633ab0fe81709b0a94af9fa8b21fb506f2a6cc WatchSource:0}: Error finding container 566dfa24242e3db32d38516025633ab0fe81709b0a94af9fa8b21fb506f2a6cc: Status 404 returned error can't find the container with id 566dfa24242e3db32d38516025633ab0fe81709b0a94af9fa8b21fb506f2a6cc Nov 02 20:25:45.061456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:45.061378 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" podStartSLOduration=2.061361841 podStartE2EDuration="2.061361841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:45.059098321 +0000 UTC m=+1987.669750413" watchObservedRunningTime="2026-11-02 20:25:45.061361841 +0000 UTC m=+1987.672013931" Nov 02 20:25:45.167203 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:45.167166 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7"] Nov 02 20:25:45.178569 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:45.178488 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-n5zk7"] Nov 02 20:25:45.510497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:45.510468 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43cff83a-d81e-49bb-b7e1-956e820bc615" path="/var/lib/kubelet/pods/43cff83a-d81e-49bb-b7e1-956e820bc615/volumes" Nov 02 20:25:45.615995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:45.615956 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-602be312-5597fb8dbc-nmjht" podUID="dc9560d7-7263-49b9-99e4-012124e9a461" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.120:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:25:45.977382 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:45.977342 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" event={"ID":"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4","Type":"ContainerStarted","Data":"3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503"} Nov 02 20:25:45.978053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:45.977991 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" event={"ID":"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4","Type":"ContainerStarted","Data":"566dfa24242e3db32d38516025633ab0fe81709b0a94af9fa8b21fb506f2a6cc"} Nov 02 20:25:45.978053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:45.978057 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:25:46.114978 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:46.114922 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" podStartSLOduration=3.114906422 podStartE2EDuration="3.114906422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:46.108101827 +0000 UTC m=+1988.718753915" watchObservedRunningTime="2026-11-02 20:25:46.114906422 +0000 UTC m=+1988.725558510" Nov 02 20:25:48.238928 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:25:48.238847 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:25:54.619299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.618250 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl"] Nov 02 20:25:54.625296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.622556 5399 cpu_manager.go:414] "containerMap: removing container" podUID="43cff83a-d81e-49bb-b7e1-956e820bc615" containerName="istio-proxy" Nov 02 20:25:54.625296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.622586 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cff83a-d81e-49bb-b7e1-956e820bc615" containerName="istio-proxy" Nov 02 20:25:54.625296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.622855 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cff83a-d81e-49bb-b7e1-956e820bc615" containerName="istio-proxy" Nov 02 20:25:54.638676 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.629073 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:25:54.659379 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.659343 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl"] Nov 02 20:25:54.755576 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.752894 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-abc937\"" Nov 02 20:25:54.755576 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.753668 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-abc937-dockercfg-4mr2h\"" Nov 02 20:25:54.857118 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.857085 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9668e2fe-b325-44d5-8551-50e5f563a14e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl\" (UID: \"9668e2fe-b325-44d5-8551-50e5f563a14e\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:25:54.857369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.857224 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjmxq\" (UniqueName: \"kubernetes.io/projected/9668e2fe-b325-44d5-8551-50e5f563a14e-kube-api-access-qjmxq\") pod \"payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl\" (UID: \"9668e2fe-b325-44d5-8551-50e5f563a14e\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:25:54.958959 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.958861 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9668e2fe-b325-44d5-8551-50e5f563a14e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl\" (UID: \"9668e2fe-b325-44d5-8551-50e5f563a14e\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:25:54.958959 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.958966 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjmxq\" (UniqueName: \"kubernetes.io/projected/9668e2fe-b325-44d5-8551-50e5f563a14e-kube-api-access-qjmxq\") pod \"payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl\" (UID: \"9668e2fe-b325-44d5-8551-50e5f563a14e\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:25:54.959965 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:54.959926 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9668e2fe-b325-44d5-8551-50e5f563a14e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl\" (UID: \"9668e2fe-b325-44d5-8551-50e5f563a14e\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:25:55.013717 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:55.013682 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjmxq\" (UniqueName: \"kubernetes.io/projected/9668e2fe-b325-44d5-8551-50e5f563a14e-kube-api-access-qjmxq\") pod \"payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl\" (UID: \"9668e2fe-b325-44d5-8551-50e5f563a14e\") " pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:25:55.258164 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:55.258081 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:25:55.632047 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:55.632020 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl"] Nov 02 20:25:55.632515 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:25:55.632493 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9668e2fe_b325_44d5_8551_50e5f563a14e.slice/crio-9f28b0b942f7975aa1a556fc5946a84ec9c50e87538b15769016c2a2168ff768 WatchSource:0}: Error finding container 9f28b0b942f7975aa1a556fc5946a84ec9c50e87538b15769016c2a2168ff768: Status 404 returned error can't find the container with id 9f28b0b942f7975aa1a556fc5946a84ec9c50e87538b15769016c2a2168ff768 Nov 02 20:25:55.981002 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:55.980915 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:25:56.069492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:56.069456 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" event={"ID":"9668e2fe-b325-44d5-8551-50e5f563a14e","Type":"ContainerStarted","Data":"c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317"} Nov 02 20:25:56.069662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:56.069518 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" event={"ID":"9668e2fe-b325-44d5-8551-50e5f563a14e","Type":"ContainerStarted","Data":"9f28b0b942f7975aa1a556fc5946a84ec9c50e87538b15769016c2a2168ff768"} Nov 02 20:25:56.069662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:56.069551 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:25:56.192565 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:56.191893 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" podStartSLOduration=2.191873264 podStartE2EDuration="2.191873264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:56.191847508 +0000 UTC m=+1998.802499600" watchObservedRunningTime="2026-11-02 20:25:56.191873264 +0000 UTC m=+1998.802525357" Nov 02 20:25:56.996676 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:25:56.996638 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:26:03.325875 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:03.325840 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:26:07.080834 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:07.080803 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:26:07.571972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:07.571936 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f"] Nov 02 20:26:07.572174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:07.572135 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" containerID="cri-o://f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e" gracePeriod=30 Nov 02 20:26:08.516163 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:08.516128 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk"] Nov 02 20:26:08.516556 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:08.516336 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" containerID="cri-o://b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d" gracePeriod=30 Nov 02 20:26:09.773962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:09.773921 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:26:09.774499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:09.773985 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:26:10.576305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:10.576251 5399 patch_prober.go:29] interesting pod/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:26:10.576490 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:10.576337 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:26:12.863889 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.863862 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:26:12.894452 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894416 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-socket\") pod \"64eae2e1-db77-4773-89f6-2ab7cee55116\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " Nov 02 20:26:12.894623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894504 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-certs\") pod \"64eae2e1-db77-4773-89f6-2ab7cee55116\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " Nov 02 20:26:12.894623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894545 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-data\") pod \"64eae2e1-db77-4773-89f6-2ab7cee55116\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " Nov 02 20:26:12.894623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894612 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-token\" (UniqueName: \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-token\") pod \"64eae2e1-db77-4773-89f6-2ab7cee55116\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " Nov 02 20:26:12.894862 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894653 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-podinfo\") pod \"64eae2e1-db77-4773-89f6-2ab7cee55116\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " Nov 02 20:26:12.894862 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894699 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-socket" pod "64eae2e1-db77-4773-89f6-2ab7cee55116" (UID: "64eae2e1-db77-4773-89f6-2ab7cee55116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:12.894862 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894776 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-kube-api-access-h6h5g\" (UniqueName: \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-kube-api-access-h6h5g\") pod \"64eae2e1-db77-4773-89f6-2ab7cee55116\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " Nov 02 20:26:12.894862 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894793 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-certs" pod "64eae2e1-db77-4773-89f6-2ab7cee55116" (UID: "64eae2e1-db77-4773-89f6-2ab7cee55116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:12.894862 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894810 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-credential-socket\") pod \"64eae2e1-db77-4773-89f6-2ab7cee55116\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " Nov 02 20:26:12.895103 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894869 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-data" pod "64eae2e1-db77-4773-89f6-2ab7cee55116" (UID: "64eae2e1-db77-4773-89f6-2ab7cee55116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:12.895103 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894903 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-envoy\") pod \"64eae2e1-db77-4773-89f6-2ab7cee55116\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " Nov 02 20:26:12.895103 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.894926 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64eae2e1-db77-4773-89f6-2ab7cee55116-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eae2e1-db77-4773-89f6-2ab7cee55116-istiod-ca-cert\") pod \"64eae2e1-db77-4773-89f6-2ab7cee55116\" (UID: \"64eae2e1-db77-4773-89f6-2ab7cee55116\") " Nov 02 20:26:12.895325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.895265 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-credential-socket" pod "64eae2e1-db77-4773-89f6-2ab7cee55116" (UID: "64eae2e1-db77-4773-89f6-2ab7cee55116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:12.895407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.895370 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64eae2e1-db77-4773-89f6-2ab7cee55116-istiod-ca-cert" pod "64eae2e1-db77-4773-89f6-2ab7cee55116" (UID: "64eae2e1-db77-4773-89f6-2ab7cee55116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:26:12.895847 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.895821 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:12.895942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.895853 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:12.895942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.895870 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:12.895942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.895886 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:12.895942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.895901 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eae2e1-db77-4773-89f6-2ab7cee55116-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:12.897000 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.896955 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-token" pod "64eae2e1-db77-4773-89f6-2ab7cee55116" (UID: "64eae2e1-db77-4773-89f6-2ab7cee55116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:12.897158 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.897137 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-kube-api-access-h6h5g" pod "64eae2e1-db77-4773-89f6-2ab7cee55116" (UID: "64eae2e1-db77-4773-89f6-2ab7cee55116"). InnerVolumeSpecName "kube-api-access-h6h5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:12.897206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.897145 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-envoy" pod "64eae2e1-db77-4773-89f6-2ab7cee55116" (UID: "64eae2e1-db77-4773-89f6-2ab7cee55116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:12.897206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.897140 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-podinfo" pod "64eae2e1-db77-4773-89f6-2ab7cee55116" (UID: "64eae2e1-db77-4773-89f6-2ab7cee55116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:26:12.997243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.997202 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6h5g\" (UniqueName: \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-kube-api-access-h6h5g\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:12.997243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.997233 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:12.997243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.997243 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:12.997487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:12.997254 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eae2e1-db77-4773-89f6-2ab7cee55116-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:13.206699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.206614 5399 generic.go:356] "Generic (PLEG): container finished" podID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerID="f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e" exitCode=0 Nov 02 20:26:13.206699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.206673 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" Nov 02 20:26:13.206907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.206692 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" event={"ID":"64eae2e1-db77-4773-89f6-2ab7cee55116","Type":"ContainerDied","Data":"f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e"} Nov 02 20:26:13.206907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.206754 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f" event={"ID":"64eae2e1-db77-4773-89f6-2ab7cee55116","Type":"ContainerDied","Data":"f3a0f69587e8eac66309099994a05d624efba7cb196d7166d96ef011aff2dfc4"} Nov 02 20:26:13.206907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.206785 5399 scope.go:122] "RemoveContainer" containerID="f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e" Nov 02 20:26:13.219088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.219070 5399 scope.go:122] "RemoveContainer" containerID="f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e" Nov 02 20:26:13.219403 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:26:13.219384 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e\": container with ID starting with f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e not found: ID does not exist" containerID="f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e" Nov 02 20:26:13.219477 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.219411 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e"} err="failed to get container status \"f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e\": rpc error: code = NotFound desc = could not find container \"f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e\": container with ID starting with f1400f7a6e15d9ece732fd4fec705ca113173179c2702f72c3f5dcea88afcc5e not found: ID does not exist" Nov 02 20:26:13.361143 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.361108 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f"] Nov 02 20:26:13.364906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.364865 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-fn99f"] Nov 02 20:26:13.510408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.510335 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" path="/var/lib/kubelet/pods/64eae2e1-db77-4773-89f6-2ab7cee55116/volumes" Nov 02 20:26:13.810144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.810120 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:26:13.908541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.908504 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-credential-socket\") pod \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " Nov 02 20:26:13.909050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.908585 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-certs\") pod \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " Nov 02 20:26:13.909050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.908648 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-podinfo\") pod \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " Nov 02 20:26:13.909050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.908697 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istiod-ca-cert\") pod \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " Nov 02 20:26:13.909050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.908815 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-envoy\") pod \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " Nov 02 20:26:13.909050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.908814 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-credential-socket" pod "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" (UID: "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:13.909050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.908927 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-socket\") pod \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " Nov 02 20:26:13.909050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.908957 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-kube-api-access-jpwpb\" (UniqueName: \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-kube-api-access-jpwpb\") pod \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " Nov 02 20:26:13.909050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.908979 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-certs" pod "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" (UID: "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:13.909525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.909217 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-data\") pod \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " Nov 02 20:26:13.909525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.909255 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-token\" (UniqueName: \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-token\") pod \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\" (UID: \"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab\") " Nov 02 20:26:13.909525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.909365 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-socket" pod "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" (UID: "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:13.909678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.909581 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-data" pod "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" (UID: "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:13.909748 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.909723 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istiod-ca-cert" pod "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" (UID: "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:26:13.910200 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.910167 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:13.910391 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.910201 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:13.910391 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.910216 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:13.910391 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.910229 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:13.910391 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.910242 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:13.912747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.912711 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-podinfo" pod "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" (UID: "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:26:13.913077 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.913041 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-token" pod "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" (UID: "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:13.913077 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.913057 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-envoy" pod "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" (UID: "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:13.914005 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:13.913980 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-kube-api-access-jpwpb" pod "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" (UID: "6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab"). InnerVolumeSpecName "kube-api-access-jpwpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:14.012051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.012010 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpwpb\" (UniqueName: \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-kube-api-access-jpwpb\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:14.012051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.012047 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:14.012266 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.012061 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:14.012266 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.012073 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:14.217440 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.217403 5399 generic.go:356] "Generic (PLEG): container finished" podID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerID="b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d" exitCode=0 Nov 02 20:26:14.217661 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.217457 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" Nov 02 20:26:14.217661 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.217487 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" event={"ID":"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab","Type":"ContainerDied","Data":"b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d"} Nov 02 20:26:14.217661 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.217555 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk" event={"ID":"6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab","Type":"ContainerDied","Data":"e697831be0262ca5015706fa8106f05ee08b9119a7fe864b231253fc883d14cc"} Nov 02 20:26:14.217661 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.217587 5399 scope.go:122] "RemoveContainer" containerID="b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d" Nov 02 20:26:14.229365 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.229346 5399 scope.go:122] "RemoveContainer" containerID="b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d" Nov 02 20:26:14.229636 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:26:14.229615 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d\": container with ID starting with b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d not found: ID does not exist" containerID="b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d" Nov 02 20:26:14.229716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.229646 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d"} err="failed to get container status \"b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d\": rpc error: code = NotFound desc = could not find container \"b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d\": container with ID starting with b80e7edb7c2f812707381e72f22c54b08fd0ea1ef8074b13f8461a8052d7b63d not found: ID does not exist" Nov 02 20:26:14.375654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.375614 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk"] Nov 02 20:26:14.380098 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:14.380074 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-6tfbk"] Nov 02 20:26:15.510602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:15.510571 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" path="/var/lib/kubelet/pods/6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab/volumes" Nov 02 20:26:18.005209 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.005170 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn"] Nov 02 20:26:18.005922 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.005906 5399 cpu_manager.go:414] "containerMap: removing container" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" Nov 02 20:26:18.005974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.005923 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" Nov 02 20:26:18.005974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.005942 5399 cpu_manager.go:414] "containerMap: removing container" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" Nov 02 20:26:18.005974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.005947 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" Nov 02 20:26:18.006086 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.006047 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="64eae2e1-db77-4773-89f6-2ab7cee55116" containerName="istio-proxy" Nov 02 20:26:18.006086 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.006058 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e242ab5-a0fc-47c2-ace5-ad38ab49b0ab" containerName="istio-proxy" Nov 02 20:26:18.011236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.011218 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.063090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.063054 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn"] Nov 02 20:26:18.074465 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.074431 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-metrics-certs\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.074635 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.074594 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6dj\" (UniqueName: \"kubernetes.io/projected/e92863ab-b872-43ad-a152-709196731a86-kube-api-access-8m6dj\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.074739 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.074722 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-maas-api-tls\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.074791 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.074779 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92863ab-b872-43ad-a152-709196731a86-openshift-service-ca\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.103784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.103744 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-81e58f-serving-cert\"" Nov 02 20:26:18.176764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.176733 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92863ab-b872-43ad-a152-709196731a86-openshift-service-ca\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.176928 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.176883 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-metrics-certs\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.176972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.176948 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6dj\" (UniqueName: \"kubernetes.io/projected/e92863ab-b872-43ad-a152-709196731a86-kube-api-access-8m6dj\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.177048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.177034 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-maas-api-tls\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.177169 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:26:18.177155 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-serving-cert: secret "maas-api-e2e-api-a-81e58f-serving-cert" not found Nov 02 20:26:18.177228 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:26:18.177219 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-maas-api-tls podName:e92863ab-b872-43ad-a152-709196731a86 nodeName:}" failed. No retries permitted until 2026-11-02 20:26:18.677202421 +0000 UTC m=+2021.287854509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-maas-api-tls") pod "maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" (UID: "e92863ab-b872-43ad-a152-709196731a86") : secret "maas-api-e2e-api-a-81e58f-serving-cert" not found Nov 02 20:26:18.177388 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.177367 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92863ab-b872-43ad-a152-709196731a86-openshift-service-ca\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.179502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.179473 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-metrics-certs\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.230621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.230587 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6dj\" (UniqueName: \"kubernetes.io/projected/e92863ab-b872-43ad-a152-709196731a86-kube-api-access-8m6dj\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.237633 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:26:18.237596 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:26:18.688084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.688038 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-maas-api-tls\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.690657 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.690630 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-maas-api-tls\") pod \"maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:18.927333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:18.927294 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:19.298395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:19.298365 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn"] Nov 02 20:26:19.298819 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:26:19.298428 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92863ab_b872_43ad_a152_709196731a86.slice/crio-69b019d8b779eecce1731b5dcd77d6c4081e9373d863f398598c424445977c47 WatchSource:0}: Error finding container 69b019d8b779eecce1731b5dcd77d6c4081e9373d863f398598c424445977c47: Status 404 returned error can't find the container with id 69b019d8b779eecce1731b5dcd77d6c4081e9373d863f398598c424445977c47 Nov 02 20:26:19.300037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:19.300020 5399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:26:20.284619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:20.284572 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" event={"ID":"e92863ab-b872-43ad-a152-709196731a86","Type":"ContainerStarted","Data":"0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0"} Nov 02 20:26:20.284792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:20.284637 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" event={"ID":"e92863ab-b872-43ad-a152-709196731a86","Type":"ContainerStarted","Data":"69b019d8b779eecce1731b5dcd77d6c4081e9373d863f398598c424445977c47"} Nov 02 20:26:20.284792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:20.284668 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:20.391155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:20.391041 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" podStartSLOduration=2.6064166159999997 podStartE2EDuration="3.391026535s" totalImagesPullingTime="784.609919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:20.390769792 +0000 UTC m=+2023.001421884" watchObservedRunningTime="2026-11-02 20:26:20.391026535 +0000 UTC m=+2023.001678625" Nov 02 20:26:26.300359 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:26.300325 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:26:27.167210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.167172 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48"] Nov 02 20:26:27.172104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.172080 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.224373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.224338 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48"] Nov 02 20:26:27.229896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.229863 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99461eb1-62b6-4369-8aec-3cb288042a0b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.230047 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.230017 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwg7r\" (UniqueName: \"kubernetes.io/projected/99461eb1-62b6-4369-8aec-3cb288042a0b-kube-api-access-vwg7r\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.230381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.230356 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-maas-api-tls\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.230470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.230397 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-metrics-certs\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.265378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.265344 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-429b45-serving-cert\"" Nov 02 20:26:27.332239 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.332200 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-maas-api-tls\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.332641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.332251 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-metrics-certs\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.332641 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:26:27.332351 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-serving-cert: secret "maas-api-e2e-inf-b-429b45-serving-cert" not found Nov 02 20:26:27.332641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.332367 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99461eb1-62b6-4369-8aec-3cb288042a0b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.332641 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:26:27.332419 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-maas-api-tls podName:99461eb1-62b6-4369-8aec-3cb288042a0b nodeName:}" failed. No retries permitted until 2026-11-02 20:26:27.832403102 +0000 UTC m=+2030.443055171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-maas-api-tls") pod "maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" (UID: "99461eb1-62b6-4369-8aec-3cb288042a0b") : secret "maas-api-e2e-inf-b-429b45-serving-cert" not found Nov 02 20:26:27.332641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.332490 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwg7r\" (UniqueName: \"kubernetes.io/projected/99461eb1-62b6-4369-8aec-3cb288042a0b-kube-api-access-vwg7r\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.333076 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.333048 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99461eb1-62b6-4369-8aec-3cb288042a0b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.335096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.335072 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-metrics-certs\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.385703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.385672 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwg7r\" (UniqueName: \"kubernetes.io/projected/99461eb1-62b6-4369-8aec-3cb288042a0b-kube-api-access-vwg7r\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.841073 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.841013 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-maas-api-tls\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:27.843602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:27.843571 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-maas-api-tls\") pod \"maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:28.087489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:28.087446 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:28.465166 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:28.465134 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48"] Nov 02 20:26:28.465583 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:26:28.465398 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99461eb1_62b6_4369_8aec_3cb288042a0b.slice/crio-a03f6a8cea933f1e0ce322c7d3a207192cc9888732dab9b410879ed6a0d9955d WatchSource:0}: Error finding container a03f6a8cea933f1e0ce322c7d3a207192cc9888732dab9b410879ed6a0d9955d: Status 404 returned error can't find the container with id a03f6a8cea933f1e0ce322c7d3a207192cc9888732dab9b410879ed6a0d9955d Nov 02 20:26:29.368330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:29.368297 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" event={"ID":"99461eb1-62b6-4369-8aec-3cb288042a0b","Type":"ContainerStarted","Data":"a03f6a8cea933f1e0ce322c7d3a207192cc9888732dab9b410879ed6a0d9955d"} Nov 02 20:26:30.377688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:30.377648 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" event={"ID":"99461eb1-62b6-4369-8aec-3cb288042a0b","Type":"ContainerStarted","Data":"478b1cf24bebf8b12c9480cd4441ba777dfd6d0e7f1c4de4d9cc063504ba94e7"} Nov 02 20:26:30.378097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:30.377721 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:30.487600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:30.487541 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" podStartSLOduration=2.594881162 podStartE2EDuration="3.487527648s" totalImagesPullingTime="892.646486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:30.483679912 +0000 UTC m=+2033.094332006" watchObservedRunningTime="2026-11-02 20:26:30.487527648 +0000 UTC m=+2033.098179739" Nov 02 20:26:36.393270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:36.393241 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:26:48.322849 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:26:48.319994 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:26:49.666830 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:26:49.666794 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:27:03.545124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.545093 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl"] Nov 02 20:27:03.549641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.549624 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.602850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.602809 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl"] Nov 02 20:27:03.642129 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.642092 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-metrics-certs\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.642327 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.642243 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-openshift-service-ca\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.642529 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.642502 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-maas-api-tls\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.642589 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.642564 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2cph\" (UniqueName: \"kubernetes.io/projected/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-kube-api-access-c2cph\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.642868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.642852 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d5a51f-serving-cert\"" Nov 02 20:27:03.744135 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.744101 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-openshift-service-ca\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.744345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.744309 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-maas-api-tls\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.744429 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.744355 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2cph\" (UniqueName: \"kubernetes.io/projected/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-kube-api-access-c2cph\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.744492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.744442 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-metrics-certs\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.744492 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:27:03.744461 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-serving-cert: secret "maas-api-e2e-api-a-d5a51f-serving-cert" not found Nov 02 20:27:03.744609 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:27:03.744525 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-maas-api-tls podName:cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f nodeName:}" failed. No retries permitted until 2026-11-02 20:27:04.244503055 +0000 UTC m=+2066.855155125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-maas-api-tls") pod "maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" (UID: "cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f") : secret "maas-api-e2e-api-a-d5a51f-serving-cert" not found Nov 02 20:27:03.744922 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.744899 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-openshift-service-ca\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.746860 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.746842 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-metrics-certs\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:03.801750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:03.801682 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2cph\" (UniqueName: \"kubernetes.io/projected/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-kube-api-access-c2cph\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:04.252894 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:04.252858 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-maas-api-tls\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:04.255374 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:04.255352 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-maas-api-tls\") pod \"maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:04.463395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:04.463362 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:04.839153 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:27:04.839122 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd80cb29_3e01_48eb_9acf_7ca95bc5ad8f.slice/crio-fdc64f8353b317ff5f82df039fc3bdaa490c45f213165a0d8f30ebddc4924d96 WatchSource:0}: Error finding container fdc64f8353b317ff5f82df039fc3bdaa490c45f213165a0d8f30ebddc4924d96: Status 404 returned error can't find the container with id fdc64f8353b317ff5f82df039fc3bdaa490c45f213165a0d8f30ebddc4924d96 Nov 02 20:27:04.839650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:04.839380 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl"] Nov 02 20:27:05.710668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:05.710637 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" event={"ID":"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f","Type":"ContainerStarted","Data":"fdc64f8353b317ff5f82df039fc3bdaa490c45f213165a0d8f30ebddc4924d96"} Nov 02 20:27:06.723723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:06.723683 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" event={"ID":"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f","Type":"ContainerStarted","Data":"3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df"} Nov 02 20:27:06.724206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:06.723786 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:06.833784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:06.833727 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" podStartSLOduration=2.978695752 podStartE2EDuration="3.833710095s" totalImagesPullingTime="855.014343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:06.831336022 +0000 UTC m=+2069.441988114" watchObservedRunningTime="2026-11-02 20:27:06.833710095 +0000 UTC m=+2069.444362188" Nov 02 20:27:12.738047 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:12.738010 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:27:18.240146 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:27:18.240050 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:27:30.815022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:30.814988 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl"] Nov 02 20:27:30.819380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:30.819361 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:30.873119 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:30.873090 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl"] Nov 02 20:27:30.914430 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:30.914391 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-abc937-serving-cert\"" Nov 02 20:27:30.973785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:30.973748 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76601a34-8e82-40c5-bc27-7960292e3183-openshift-service-ca\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:30.974210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:30.973825 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-maas-api-tls\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:30.974210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:30.973974 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-metrics-certs\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:30.974210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:30.974068 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw8vk\" (UniqueName: \"kubernetes.io/projected/76601a34-8e82-40c5-bc27-7960292e3183-kube-api-access-tw8vk\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:31.083213 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:31.082244 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76601a34-8e82-40c5-bc27-7960292e3183-openshift-service-ca\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:31.083213 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:31.082345 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-maas-api-tls\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:31.083213 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:31.082479 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-metrics-certs\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:31.083213 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:31.082569 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw8vk\" (UniqueName: \"kubernetes.io/projected/76601a34-8e82-40c5-bc27-7960292e3183-kube-api-access-tw8vk\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:31.085716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:31.084881 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76601a34-8e82-40c5-bc27-7960292e3183-openshift-service-ca\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:31.085716 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:27:31.085024 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-serving-cert: secret "maas-api-e2e-api-a-abc937-serving-cert" not found Nov 02 20:27:31.085716 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:27:31.085112 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-maas-api-tls podName:76601a34-8e82-40c5-bc27-7960292e3183 nodeName:}" failed. No retries permitted until 2026-11-02 20:27:31.585091371 +0000 UTC m=+2094.195743448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-maas-api-tls") pod "maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" (UID: "76601a34-8e82-40c5-bc27-7960292e3183") : secret "maas-api-e2e-api-a-abc937-serving-cert" not found Nov 02 20:27:31.092602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:31.092570 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-metrics-certs\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:31.141328 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:31.141297 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw8vk\" (UniqueName: \"kubernetes.io/projected/76601a34-8e82-40c5-bc27-7960292e3183-kube-api-access-tw8vk\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:31.590556 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:31.590521 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-maas-api-tls\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:31.592972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:31.592943 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-maas-api-tls\") pod \"maas-api-e2e-api-a-abc937-59db9486d4-rzwtl\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:31.733469 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:31.733440 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:32.101709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:32.101674 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl"] Nov 02 20:27:32.102188 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:27:32.102162 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76601a34_8e82_40c5_bc27_7960292e3183.slice/crio-4e9f4b478139990ee3d42f344c8539003abfeef5109d4e9778b966c56ce33fc5 WatchSource:0}: Error finding container 4e9f4b478139990ee3d42f344c8539003abfeef5109d4e9778b966c56ce33fc5: Status 404 returned error can't find the container with id 4e9f4b478139990ee3d42f344c8539003abfeef5109d4e9778b966c56ce33fc5 Nov 02 20:27:32.929144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:32.929111 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" event={"ID":"76601a34-8e82-40c5-bc27-7960292e3183","Type":"ContainerStarted","Data":"4e9f4b478139990ee3d42f344c8539003abfeef5109d4e9778b966c56ce33fc5"} Nov 02 20:27:33.938483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:33.938450 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" event={"ID":"76601a34-8e82-40c5-bc27-7960292e3183","Type":"ContainerStarted","Data":"d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c"} Nov 02 20:27:33.938895 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:33.938511 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:34.049051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:34.048996 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" podStartSLOduration=3.233451805 podStartE2EDuration="4.048981578s" totalImagesPullingTime="815.529773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:34.045080692 +0000 UTC m=+2096.655732783" watchObservedRunningTime="2026-11-02 20:27:34.048981578 +0000 UTC m=+2096.659633668" Nov 02 20:27:39.952716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:39.952684 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:27:43.182431 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:43.182397 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:27:48.287325 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:27:48.287271 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:27:55.025090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.023462 5399 patch_prober.go:29] interesting pod/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": read tcp 10.133.0.2:47960->10.133.0.132:15021: read: connection reset by peer" start-of-body= Nov 02 20:27:55.025090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.023544 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": read tcp 10.133.0.2:47960->10.133.0.132:15021: read: connection reset by peer" Nov 02 20:27:55.161866 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.161828 5399 generic.go:356] "Generic (PLEG): container finished" podID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerID="01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2" exitCode=0 Nov 02 20:27:55.162051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.161899 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" event={"ID":"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5","Type":"ContainerDied","Data":"01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2"} Nov 02 20:27:55.162507 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.162487 5399 scope.go:122] "RemoveContainer" containerID="01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2" Nov 02 20:27:55.164851 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.164817 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:55.164932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.164904 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:55.164970 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.164948 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:55.166345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.166018 5399 generic.go:356] "Generic (PLEG): container finished" podID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerID="ddfa6626bc66e0a91a179e1cf437117f535630e3439013fac48a44691d092886" exitCode=0 Nov 02 20:27:55.166345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.166122 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" event={"ID":"81a960d8-7c24-462a-9a0d-e1205a4ee579","Type":"ContainerDied","Data":"ddfa6626bc66e0a91a179e1cf437117f535630e3439013fac48a44691d092886"} Nov 02 20:27:55.166542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.166525 5399 scope.go:122] "RemoveContainer" containerID="ddfa6626bc66e0a91a179e1cf437117f535630e3439013fac48a44691d092886" Nov 02 20:27:55.168902 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.168873 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:55.169126 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.168940 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:55.169126 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:55.168980 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:56.184165 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:56.184117 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" event={"ID":"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5","Type":"ContainerStarted","Data":"6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5"} Nov 02 20:27:56.190352 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:56.190307 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" event={"ID":"81a960d8-7c24-462a-9a0d-e1205a4ee579","Type":"ContainerStarted","Data":"6bb13825239cad8328bf4083b31c65462a09ba4a20e3a29b02f8f88492e1cd6e"} Nov 02 20:27:56.262385 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:56.262344 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:27:56.262607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:56.262578 5399 patch_prober.go:29] interesting pod/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:27:56.262667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:56.262648 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:27:56.585519 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:56.585479 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:27:56.586721 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:56.586684 5399 patch_prober.go:29] interesting pod/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:27:56.586843 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:56.586736 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 20:27:57.270016 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:57.269977 5399 patch_prober.go:29] interesting pod/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:27:57.270645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:57.270611 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:27:57.549473 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:57.549374 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:27:57.585518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:57.585488 5399 patch_prober.go:29] interesting pod/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:27:57.585702 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:57.585537 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 20:27:58.275301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:58.274766 5399 patch_prober.go:29] interesting pod/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:27:58.275301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:58.274828 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:27:58.585459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:58.585418 5399 patch_prober.go:29] interesting pod/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:27:58.585671 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:58.585483 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 20:27:59.266302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:59.262731 5399 patch_prober.go:29] interesting pod/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:27:59.266302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:59.262807 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:27:59.585020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:59.584985 5399 patch_prober.go:29] interesting pod/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:27:59.585451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:27:59.585041 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 20:28:00.264921 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:00.264882 5399 patch_prober.go:29] interesting pod/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:28:00.265214 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:00.265187 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:28:00.584891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:00.584851 5399 patch_prober.go:29] interesting pod/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:28:00.585118 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:00.584915 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 20:28:01.265177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:01.265141 5399 patch_prober.go:29] interesting pod/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 20:28:01.265503 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:01.265470 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 20:28:01.589660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:01.587769 5399 patch_prober.go:29] interesting pod/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:28:01.589660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:01.587937 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 20:28:02.267155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:02.267120 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:28:02.267648 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:02.267626 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:28:02.268233 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:02.268217 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:28:02.590824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:02.590792 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:28:02.592195 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:02.592171 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:28:07.654499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.654461 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd"] Nov 02 20:28:07.659222 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.659201 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.707493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.707458 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-602be312-openshift-default-dockercfg-4c2qs\"" Nov 02 20:28:07.760780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.760737 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.760974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.760791 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.760974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.760842 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.760974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.760909 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.761153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.760999 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.761153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.761033 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.761153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.761075 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvlr7\" (UniqueName: \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-kube-api-access-hvlr7\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.761321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.761194 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.761321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.761260 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.761780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.761750 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd"] Nov 02 20:28:07.862492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.862453 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.862697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.862498 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.862697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.862584 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.862697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.862639 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.862697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.862680 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvlr7\" (UniqueName: \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-kube-api-access-hvlr7\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.862930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.862792 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.862930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.862855 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.863075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.863049 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.863158 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.863076 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.863158 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.863060 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.863158 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.863154 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.863380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.863158 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.863443 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.863416 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.863755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.863730 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.865067 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.865045 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.865432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.865414 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.915847 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.915772 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.915847 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.915813 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvlr7\" (UniqueName: \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-kube-api-access-hvlr7\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:07.973341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:07.973253 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:08.418377 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:08.418345 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd"] Nov 02 20:28:08.418849 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:28:08.418825 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc859550_a1bd_4c32_96ae_1c22edd82f6d.slice/crio-33ac5a2b4a166019dee9b404717b431c767a69c69576f0bcfc147a87a096b95a WatchSource:0}: Error finding container 33ac5a2b4a166019dee9b404717b431c767a69c69576f0bcfc147a87a096b95a: Status 404 returned error can't find the container with id 33ac5a2b4a166019dee9b404717b431c767a69c69576f0bcfc147a87a096b95a Nov 02 20:28:08.421576 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:08.421543 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:08.421664 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:08.421610 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:08.421664 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:08.421639 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:09.368145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:09.368105 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" event={"ID":"dc859550-a1bd-4c32-96ae-1c22edd82f6d","Type":"ContainerStarted","Data":"925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403"} Nov 02 20:28:09.368704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:09.368246 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" event={"ID":"dc859550-a1bd-4c32-96ae-1c22edd82f6d","Type":"ContainerStarted","Data":"33ac5a2b4a166019dee9b404717b431c767a69c69576f0bcfc147a87a096b95a"} Nov 02 20:28:09.789347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:09.789229 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" podStartSLOduration=2.789213862 podStartE2EDuration="2.789213862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:09.551693305 +0000 UTC m=+2132.162345421" watchObservedRunningTime="2026-11-02 20:28:09.789213862 +0000 UTC m=+2132.399865954" Nov 02 20:28:09.791546 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:09.791520 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh"] Nov 02 20:28:09.973767 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:09.973732 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:09.975598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:09.975574 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 20:28:09.975717 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:09.975628 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" podUID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 20:28:10.974573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:10.974536 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 20:28:10.975049 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:10.974604 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" podUID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 20:28:11.974630 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:11.974591 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 20:28:11.975035 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:11.974660 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" podUID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 20:28:12.979134 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:12.979093 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 20:28:12.979675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:12.979180 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" podUID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 20:28:13.980298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:13.977317 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 02 20:28:13.980298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:13.977387 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" podUID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 02 20:28:14.791863 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:14.791819 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" podUID="52bd4f76-0054-4157-8ab2-f801a14b06ea" containerName="payload-processing" containerID="cri-o://eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0" gracePeriod=25 Nov 02 20:28:15.120314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.118620 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:28:15.120314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.119602 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:15.120314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.120027 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:15.121044 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.121025 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:28:15.229420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.229389 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bd4f76-0054-4157-8ab2-f801a14b06ea-kube-api-access-d4prd\" (UniqueName: \"kubernetes.io/projected/52bd4f76-0054-4157-8ab2-f801a14b06ea-kube-api-access-d4prd\") pod \"52bd4f76-0054-4157-8ab2-f801a14b06ea\" (UID: \"52bd4f76-0054-4157-8ab2-f801a14b06ea\") " Nov 02 20:28:15.229709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.229691 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bd4f76-0054-4157-8ab2-f801a14b06ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bd4f76-0054-4157-8ab2-f801a14b06ea-plugins-config-volume\") pod \"52bd4f76-0054-4157-8ab2-f801a14b06ea\" (UID: \"52bd4f76-0054-4157-8ab2-f801a14b06ea\") " Nov 02 20:28:15.231209 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.231177 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bd4f76-0054-4157-8ab2-f801a14b06ea-plugins-config-volume" pod "52bd4f76-0054-4157-8ab2-f801a14b06ea" (UID: "52bd4f76-0054-4157-8ab2-f801a14b06ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:15.232824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.232798 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bd4f76-0054-4157-8ab2-f801a14b06ea-kube-api-access-d4prd" pod "52bd4f76-0054-4157-8ab2-f801a14b06ea" (UID: "52bd4f76-0054-4157-8ab2-f801a14b06ea"). InnerVolumeSpecName "kube-api-access-d4prd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:15.332270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.332184 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bd4f76-0054-4157-8ab2-f801a14b06ea-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:15.332270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.332217 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4prd\" (UniqueName: \"kubernetes.io/projected/52bd4f76-0054-4157-8ab2-f801a14b06ea-kube-api-access-d4prd\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:15.500676 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.500643 5399 generic.go:356] "Generic (PLEG): container finished" podID="52bd4f76-0054-4157-8ab2-f801a14b06ea" containerID="eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0" exitCode=0 Nov 02 20:28:15.500877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.500755 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" Nov 02 20:28:15.513107 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.513076 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" event={"ID":"52bd4f76-0054-4157-8ab2-f801a14b06ea","Type":"ContainerDied","Data":"eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0"} Nov 02 20:28:15.513228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.513126 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh" event={"ID":"52bd4f76-0054-4157-8ab2-f801a14b06ea","Type":"ContainerDied","Data":"85496fb70c09064b7c43e318e101b3d39f66cca1f3676ddc60980d7d62c8e385"} Nov 02 20:28:15.513228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.513164 5399 scope.go:122] "RemoveContainer" containerID="eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0" Nov 02 20:28:15.529374 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.529348 5399 scope.go:122] "RemoveContainer" containerID="eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0" Nov 02 20:28:15.529714 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:28:15.529692 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0\": container with ID starting with eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0 not found: ID does not exist" containerID="eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0" Nov 02 20:28:15.529812 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.529723 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0"} err="failed to get container status \"eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0\": rpc error: code = NotFound desc = could not find container \"eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0\": container with ID starting with eaec0f02e27a6b35975f7abb72a459362e0e783752bdbf4628bd323eba2830f0 not found: ID does not exist" Nov 02 20:28:15.655001 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.654909 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh"] Nov 02 20:28:15.659375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:15.659345 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-81e58f-87659f99c-4lnmh"] Nov 02 20:28:17.511749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:17.511714 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bd4f76-0054-4157-8ab2-f801a14b06ea" path="/var/lib/kubelet/pods/52bd4f76-0054-4157-8ab2-f801a14b06ea/volumes" Nov 02 20:28:18.378220 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:28:18.378172 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:28:22.765035 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.764989 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp"] Nov 02 20:28:22.765965 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.765941 5399 cpu_manager.go:414] "containerMap: removing container" podUID="52bd4f76-0054-4157-8ab2-f801a14b06ea" containerName="payload-processing" Nov 02 20:28:22.766075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.765966 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bd4f76-0054-4157-8ab2-f801a14b06ea" containerName="payload-processing" Nov 02 20:28:22.766216 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.766202 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bd4f76-0054-4157-8ab2-f801a14b06ea" containerName="payload-processing" Nov 02 20:28:22.771205 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.771183 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.858316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.856448 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.858316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.856521 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.858316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.856549 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.858316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.856610 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.858316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.856652 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmdcv\" (UniqueName: \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-kube-api-access-tmdcv\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.858316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.856812 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.858316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.856876 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.858316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.857028 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.858316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.857067 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.956507 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.956460 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp"] Nov 02 20:28:22.958441 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.958406 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.958611 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.958473 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.958611 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.958507 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.958611 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.958572 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.958793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.958614 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmdcv\" (UniqueName: \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-kube-api-access-tmdcv\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.958793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.958773 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.958908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.958832 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.958999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.958979 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.958999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.958984 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-certs\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.959182 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.959025 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.959467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.959443 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-data\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.959732 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.959708 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.959983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.959961 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-credential-socket\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.960299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.960242 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istiod-ca-cert\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.965146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.965123 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-envoy\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:22.965346 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:22.965325 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-podinfo\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:23.012251 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:23.012224 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmdcv\" (UniqueName: \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-kube-api-access-tmdcv\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:23.013736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:23.013698 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-token\") pod \"e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:23.086025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:23.085986 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:23.560143 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:23.560115 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp"] Nov 02 20:28:23.560292 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:28:23.560190 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3daeb5a8_fef8_4c6d_a94c_0116902df49b.slice/crio-d4508d387b0e225ec9c5e2d2645d899414d2a745f5e7118b0b06afeecd85bf5e WatchSource:0}: Error finding container d4508d387b0e225ec9c5e2d2645d899414d2a745f5e7118b0b06afeecd85bf5e: Status 404 returned error can't find the container with id d4508d387b0e225ec9c5e2d2645d899414d2a745f5e7118b0b06afeecd85bf5e Nov 02 20:28:23.563206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:23.563175 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:23.563316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:23.563239 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:23.563316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:23.563268 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:23.569604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:23.569573 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" event={"ID":"3daeb5a8-fef8-4c6d-a94c-0116902df49b","Type":"ContainerStarted","Data":"d4508d387b0e225ec9c5e2d2645d899414d2a745f5e7118b0b06afeecd85bf5e"} Nov 02 20:28:24.580109 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:24.580062 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" event={"ID":"3daeb5a8-fef8-4c6d-a94c-0116902df49b","Type":"ContainerStarted","Data":"605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a"} Nov 02 20:28:25.086343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:25.086298 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:25.087817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:25.087788 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:28:25.087958 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:25.087843 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:28:26.089161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:26.089114 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:28:26.089612 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:26.089187 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:28:27.087169 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:27.087135 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:28:27.087376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:27.087184 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:28:28.086927 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:28.086890 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:28:28.087363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:28.086954 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:28:29.087409 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:29.087356 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:28:29.087928 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:29.087423 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:28:30.052303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:30.051838 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" podStartSLOduration=8.051813377 podStartE2EDuration="8.051813377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:24.751788029 +0000 UTC m=+2147.362440121" watchObservedRunningTime="2026-11-02 20:28:30.051813377 +0000 UTC m=+2152.662465469" Nov 02 20:28:30.055415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:30.055382 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm"] Nov 02 20:28:30.086456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:30.086422 5399 patch_prober.go:29] interesting pod/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 20:28:30.086623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:30.086483 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 20:28:30.092652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:30.092620 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575830cb-4csqm"] Nov 02 20:28:31.023714 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:31.023676 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd"] Nov 02 20:28:31.091416 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:31.091386 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:31.091708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:31.091683 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:31.092394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:31.092372 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:28:31.137950 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:31.137914 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f"] Nov 02 20:28:31.511867 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:31.511829 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4898325-a1c0-4353-b407-4cc188f2f963" path="/var/lib/kubelet/pods/f4898325-a1c0-4353-b407-4cc188f2f963/volumes" Nov 02 20:28:36.024857 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.024799 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" podUID="3313660b-c048-4c33-8524-30f760d0b1e0" containerName="payload-processing" containerID="cri-o://71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1" gracePeriod=25 Nov 02 20:28:36.139183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.139110 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" podUID="ec25bbd4-d01c-48f2-840a-81e159db4939" containerName="payload-pre-processing" containerID="cri-o://c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a" gracePeriod=25 Nov 02 20:28:36.324489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.324462 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:28:36.420364 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.419699 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3313660b-c048-4c33-8524-30f760d0b1e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3313660b-c048-4c33-8524-30f760d0b1e0-plugins-config-volume\") pod \"3313660b-c048-4c33-8524-30f760d0b1e0\" (UID: \"3313660b-c048-4c33-8524-30f760d0b1e0\") " Nov 02 20:28:36.420364 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.419869 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3313660b-c048-4c33-8524-30f760d0b1e0-kube-api-access-45ckn\" (UniqueName: \"kubernetes.io/projected/3313660b-c048-4c33-8524-30f760d0b1e0-kube-api-access-45ckn\") pod \"3313660b-c048-4c33-8524-30f760d0b1e0\" (UID: \"3313660b-c048-4c33-8524-30f760d0b1e0\") " Nov 02 20:28:36.420364 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.420131 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3313660b-c048-4c33-8524-30f760d0b1e0-plugins-config-volume" pod "3313660b-c048-4c33-8524-30f760d0b1e0" (UID: "3313660b-c048-4c33-8524-30f760d0b1e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:36.420796 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.420757 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3313660b-c048-4c33-8524-30f760d0b1e0-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:36.422226 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.422200 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3313660b-c048-4c33-8524-30f760d0b1e0-kube-api-access-45ckn" pod "3313660b-c048-4c33-8524-30f760d0b1e0" (UID: "3313660b-c048-4c33-8524-30f760d0b1e0"). InnerVolumeSpecName "kube-api-access-45ckn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:36.448407 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.448379 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:28:36.521429 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.521401 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec25bbd4-d01c-48f2-840a-81e159db4939-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec25bbd4-d01c-48f2-840a-81e159db4939-plugins-config-volume\") pod \"ec25bbd4-d01c-48f2-840a-81e159db4939\" (UID: \"ec25bbd4-d01c-48f2-840a-81e159db4939\") " Nov 02 20:28:36.521599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.521551 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec25bbd4-d01c-48f2-840a-81e159db4939-kube-api-access-tst4m\" (UniqueName: \"kubernetes.io/projected/ec25bbd4-d01c-48f2-840a-81e159db4939-kube-api-access-tst4m\") pod \"ec25bbd4-d01c-48f2-840a-81e159db4939\" (UID: \"ec25bbd4-d01c-48f2-840a-81e159db4939\") " Nov 02 20:28:36.521761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.521737 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec25bbd4-d01c-48f2-840a-81e159db4939-plugins-config-volume" pod "ec25bbd4-d01c-48f2-840a-81e159db4939" (UID: "ec25bbd4-d01c-48f2-840a-81e159db4939"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:36.522311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.522288 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45ckn\" (UniqueName: \"kubernetes.io/projected/3313660b-c048-4c33-8524-30f760d0b1e0-kube-api-access-45ckn\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:36.522311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.522314 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec25bbd4-d01c-48f2-840a-81e159db4939-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:36.524009 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.523986 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec25bbd4-d01c-48f2-840a-81e159db4939-kube-api-access-tst4m" pod "ec25bbd4-d01c-48f2-840a-81e159db4939" (UID: "ec25bbd4-d01c-48f2-840a-81e159db4939"). InnerVolumeSpecName "kube-api-access-tst4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:36.624005 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.623965 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tst4m\" (UniqueName: \"kubernetes.io/projected/ec25bbd4-d01c-48f2-840a-81e159db4939-kube-api-access-tst4m\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:36.725484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.725449 5399 generic.go:356] "Generic (PLEG): container finished" podID="ec25bbd4-d01c-48f2-840a-81e159db4939" containerID="c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a" exitCode=0 Nov 02 20:28:36.725658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.725496 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" Nov 02 20:28:36.725658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.725527 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" event={"ID":"ec25bbd4-d01c-48f2-840a-81e159db4939","Type":"ContainerDied","Data":"c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a"} Nov 02 20:28:36.725658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.725578 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f" event={"ID":"ec25bbd4-d01c-48f2-840a-81e159db4939","Type":"ContainerDied","Data":"5d8026217e6bd478c65741c0c7e6d1e03784df031b22b455e3034da3fe89e95a"} Nov 02 20:28:36.725658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.725600 5399 scope.go:122] "RemoveContainer" containerID="c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a" Nov 02 20:28:36.727162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.727137 5399 generic.go:356] "Generic (PLEG): container finished" podID="3313660b-c048-4c33-8524-30f760d0b1e0" containerID="71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1" exitCode=0 Nov 02 20:28:36.727333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.727181 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" Nov 02 20:28:36.727333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.727194 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" event={"ID":"3313660b-c048-4c33-8524-30f760d0b1e0","Type":"ContainerDied","Data":"71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1"} Nov 02 20:28:36.727333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.727230 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd" event={"ID":"3313660b-c048-4c33-8524-30f760d0b1e0","Type":"ContainerDied","Data":"70d6c106b18fe23a53fdfc8744d44209d2c6c25e63a392a8300d58e54980d6f1"} Nov 02 20:28:36.737596 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.737578 5399 scope.go:122] "RemoveContainer" containerID="c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a" Nov 02 20:28:36.737839 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:28:36.737823 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a\": container with ID starting with c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a not found: ID does not exist" containerID="c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a" Nov 02 20:28:36.737886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.737846 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a"} err="failed to get container status \"c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a\": rpc error: code = NotFound desc = could not find container \"c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a\": container with ID starting with c736263cc20fef7572dc591db152526053abba40eb7f28ef60dc07a26b89fd2a not found: ID does not exist" Nov 02 20:28:36.737886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.737861 5399 scope.go:122] "RemoveContainer" containerID="71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1" Nov 02 20:28:36.747754 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.747736 5399 scope.go:122] "RemoveContainer" containerID="71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1" Nov 02 20:28:36.748037 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:28:36.748015 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1\": container with ID starting with 71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1 not found: ID does not exist" containerID="71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1" Nov 02 20:28:36.748096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.748049 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1"} err="failed to get container status \"71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1\": rpc error: code = NotFound desc = could not find container \"71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1\": container with ID starting with 71f1943b0f2fa5d5b4e6871b1e80cbc10f33cc600e2a24969dc012d0255209e1 not found: ID does not exist" Nov 02 20:28:36.881429 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.881340 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f"] Nov 02 20:28:36.885643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.885620 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-eb060a-68774bfc59-r722f"] Nov 02 20:28:36.986192 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.986158 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd"] Nov 02 20:28:36.990137 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:36.990108 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-eb060a-f5d6b46f9-rcxgd"] Nov 02 20:28:37.535297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:37.534884 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3313660b-c048-4c33-8524-30f760d0b1e0" path="/var/lib/kubelet/pods/3313660b-c048-4c33-8524-30f760d0b1e0/volumes" Nov 02 20:28:37.537350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:37.535962 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec25bbd4-d01c-48f2-840a-81e159db4939" path="/var/lib/kubelet/pods/ec25bbd4-d01c-48f2-840a-81e159db4939/volumes" Nov 02 20:28:37.773956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:37.773917 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:28:41.220835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:41.220798 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr"] Nov 02 20:28:41.346655 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:41.346622 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4"] Nov 02 20:28:46.221435 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.221397 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" podUID="6f42b0a8-fac1-425f-9b79-0dba2dca2fa4" containerName="payload-processing" containerID="cri-o://3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503" gracePeriod=25 Nov 02 20:28:46.347720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.347654 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" podUID="16fdd199-f388-4ed2-abe5-2c2e5acb6a41" containerName="payload-pre-processing" containerID="cri-o://9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e" gracePeriod=25 Nov 02 20:28:46.523088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.523060 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:28:46.637786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.637750 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-kube-api-access-fxz44\" (UniqueName: \"kubernetes.io/projected/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-kube-api-access-fxz44\") pod \"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4\" (UID: \"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4\") " Nov 02 20:28:46.638116 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.637813 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-plugins-config-volume\") pod \"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4\" (UID: \"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4\") " Nov 02 20:28:46.639184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.639156 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-plugins-config-volume" pod "6f42b0a8-fac1-425f-9b79-0dba2dca2fa4" (UID: "6f42b0a8-fac1-425f-9b79-0dba2dca2fa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:46.640904 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.640870 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-kube-api-access-fxz44" pod "6f42b0a8-fac1-425f-9b79-0dba2dca2fa4" (UID: "6f42b0a8-fac1-425f-9b79-0dba2dca2fa4"). InnerVolumeSpecName "kube-api-access-fxz44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:46.645180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.645149 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:28:46.739586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.739502 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-plugins-config-volume\") pod \"16fdd199-f388-4ed2-abe5-2c2e5acb6a41\" (UID: \"16fdd199-f388-4ed2-abe5-2c2e5acb6a41\") " Nov 02 20:28:46.739762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.739656 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-kube-api-access-hgkf2\" (UniqueName: \"kubernetes.io/projected/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-kube-api-access-hgkf2\") pod \"16fdd199-f388-4ed2-abe5-2c2e5acb6a41\" (UID: \"16fdd199-f388-4ed2-abe5-2c2e5acb6a41\") " Nov 02 20:28:46.739918 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.739895 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-plugins-config-volume" pod "16fdd199-f388-4ed2-abe5-2c2e5acb6a41" (UID: "16fdd199-f388-4ed2-abe5-2c2e5acb6a41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:46.740342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.740319 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:46.740342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.740341 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:46.740502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.740357 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxz44\" (UniqueName: \"kubernetes.io/projected/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4-kube-api-access-fxz44\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:46.741740 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.741720 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-kube-api-access-hgkf2" pod "16fdd199-f388-4ed2-abe5-2c2e5acb6a41" (UID: "16fdd199-f388-4ed2-abe5-2c2e5acb6a41"). InnerVolumeSpecName "kube-api-access-hgkf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:46.814901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.814865 5399 generic.go:356] "Generic (PLEG): container finished" podID="6f42b0a8-fac1-425f-9b79-0dba2dca2fa4" containerID="3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503" exitCode=0 Nov 02 20:28:46.815103 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.814921 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" Nov 02 20:28:46.815103 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.814951 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" event={"ID":"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4","Type":"ContainerDied","Data":"3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503"} Nov 02 20:28:46.815103 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.815018 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr" event={"ID":"6f42b0a8-fac1-425f-9b79-0dba2dca2fa4","Type":"ContainerDied","Data":"566dfa24242e3db32d38516025633ab0fe81709b0a94af9fa8b21fb506f2a6cc"} Nov 02 20:28:46.815103 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.815042 5399 scope.go:122] "RemoveContainer" containerID="3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503" Nov 02 20:28:46.816839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.816815 5399 generic.go:356] "Generic (PLEG): container finished" podID="16fdd199-f388-4ed2-abe5-2c2e5acb6a41" containerID="9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e" exitCode=0 Nov 02 20:28:46.816950 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.816854 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" Nov 02 20:28:46.816950 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.816893 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" event={"ID":"16fdd199-f388-4ed2-abe5-2c2e5acb6a41","Type":"ContainerDied","Data":"9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e"} Nov 02 20:28:46.816950 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.816939 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4" event={"ID":"16fdd199-f388-4ed2-abe5-2c2e5acb6a41","Type":"ContainerDied","Data":"c50416f14a9ae0f383abf102baee1a0b3efa1e61e1a915189e90a01d3ba2c4ea"} Nov 02 20:28:46.832794 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.832775 5399 scope.go:122] "RemoveContainer" containerID="3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503" Nov 02 20:28:46.833061 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:28:46.833042 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503\": container with ID starting with 3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503 not found: ID does not exist" containerID="3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503" Nov 02 20:28:46.833133 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.833067 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503"} err="failed to get container status \"3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503\": rpc error: code = NotFound desc = could not find container \"3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503\": container with ID starting with 3dc7b674370eb886854212193508888d907bdbbeb2516da1d7a6fc09d6feb503 not found: ID does not exist" Nov 02 20:28:46.833133 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.833084 5399 scope.go:122] "RemoveContainer" containerID="9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e" Nov 02 20:28:46.842111 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.842088 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgkf2\" (UniqueName: \"kubernetes.io/projected/16fdd199-f388-4ed2-abe5-2c2e5acb6a41-kube-api-access-hgkf2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:46.842794 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.842775 5399 scope.go:122] "RemoveContainer" containerID="9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e" Nov 02 20:28:46.843068 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:28:46.843051 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e\": container with ID starting with 9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e not found: ID does not exist" containerID="9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e" Nov 02 20:28:46.843114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.843074 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e"} err="failed to get container status \"9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e\": rpc error: code = NotFound desc = could not find container \"9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e\": container with ID starting with 9aa71fb64f41f923880ab1e4715f3dfb1a1486dee9f0a90c030f71c2537b2a9e not found: ID does not exist" Nov 02 20:28:46.970795 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.970714 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4"] Nov 02 20:28:46.975208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:46.975179 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d5a51f-769b6bd95-k8pf4"] Nov 02 20:28:47.083476 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:47.083350 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr"] Nov 02 20:28:47.088688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:47.088648 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d5a51f-6f8c67f55-dcgxr"] Nov 02 20:28:47.511855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:47.511784 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16fdd199-f388-4ed2-abe5-2c2e5acb6a41" path="/var/lib/kubelet/pods/16fdd199-f388-4ed2-abe5-2c2e5acb6a41/volumes" Nov 02 20:28:47.512353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:47.512337 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f42b0a8-fac1-425f-9b79-0dba2dca2fa4" path="/var/lib/kubelet/pods/6f42b0a8-fac1-425f-9b79-0dba2dca2fa4/volumes" Nov 02 20:28:48.238474 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:28:48.238435 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:28:48.983643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:48.983608 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl"] Nov 02 20:28:48.984102 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:48.983818 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" podUID="cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f" containerName="maas-api" containerID="cri-o://3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df" gracePeriod=30 Nov 02 20:28:49.314476 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.314450 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:28:49.409298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.408973 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-kube-api-access-c2cph\" (UniqueName: \"kubernetes.io/projected/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-kube-api-access-c2cph\") pod \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " Nov 02 20:28:49.417296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.409546 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-openshift-service-ca\") pod \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " Nov 02 20:28:49.417296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.409765 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-maas-api-tls\") pod \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " Nov 02 20:28:49.417296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.409850 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-metrics-certs\") pod \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\" (UID: \"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f\") " Nov 02 20:28:49.417296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.411017 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-openshift-service-ca" pod "cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f" (UID: "cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:49.417654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.417529 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:49.419342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.417815 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-kube-api-access-c2cph" pod "cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f" (UID: "cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f"). InnerVolumeSpecName "kube-api-access-c2cph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:49.422026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.422003 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-metrics-certs" pod "cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f" (UID: "cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:49.429437 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.429407 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-maas-api-tls" pod "cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f" (UID: "cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:49.482412 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.482375 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml"] Nov 02 20:28:49.483247 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483211 5399 cpu_manager.go:414] "containerMap: removing container" podUID="6f42b0a8-fac1-425f-9b79-0dba2dca2fa4" containerName="payload-processing" Nov 02 20:28:49.483247 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483234 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f42b0a8-fac1-425f-9b79-0dba2dca2fa4" containerName="payload-processing" Nov 02 20:28:49.483566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483286 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3313660b-c048-4c33-8524-30f760d0b1e0" containerName="payload-processing" Nov 02 20:28:49.483566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483295 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3313660b-c048-4c33-8524-30f760d0b1e0" containerName="payload-processing" Nov 02 20:28:49.483566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483311 5399 cpu_manager.go:414] "containerMap: removing container" podUID="16fdd199-f388-4ed2-abe5-2c2e5acb6a41" containerName="payload-pre-processing" Nov 02 20:28:49.483566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483318 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16fdd199-f388-4ed2-abe5-2c2e5acb6a41" containerName="payload-pre-processing" Nov 02 20:28:49.483566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483347 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ec25bbd4-d01c-48f2-840a-81e159db4939" containerName="payload-pre-processing" Nov 02 20:28:49.483566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483353 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec25bbd4-d01c-48f2-840a-81e159db4939" containerName="payload-pre-processing" Nov 02 20:28:49.484059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483629 5399 cpu_manager.go:414] "containerMap: removing container" podUID="cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f" containerName="maas-api" Nov 02 20:28:49.484059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483645 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f" containerName="maas-api" Nov 02 20:28:49.484059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.483982 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f" containerName="maas-api" Nov 02 20:28:49.484059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.484006 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f42b0a8-fac1-425f-9b79-0dba2dca2fa4" containerName="payload-processing" Nov 02 20:28:49.484059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.484023 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3313660b-c048-4c33-8524-30f760d0b1e0" containerName="payload-processing" Nov 02 20:28:49.484059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.484035 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="16fdd199-f388-4ed2-abe5-2c2e5acb6a41" containerName="payload-pre-processing" Nov 02 20:28:49.484059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.484053 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec25bbd4-d01c-48f2-840a-81e159db4939" containerName="payload-pre-processing" Nov 02 20:28:49.488057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.488036 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.519031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.518994 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:49.519187 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.519033 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:49.519187 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.519050 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2cph\" (UniqueName: \"kubernetes.io/projected/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f-kube-api-access-c2cph\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:49.538165 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.538078 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:28:49.547927 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.547735 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml"] Nov 02 20:28:49.547927 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.547849 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:28:49.620291 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.620238 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc937-vv9ml\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.620475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.620406 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5rfd\" (UniqueName: \"kubernetes.io/projected/ed16245c-fd6f-458a-b6f5-07f82919b330-kube-api-access-d5rfd\") pod \"maas-api-revoke-keys-e2e-api-a-abc937-vv9ml\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.620707 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.620689 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc937-vv9ml\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.721923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.721880 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc937-vv9ml\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.722130 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.722018 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5rfd\" (UniqueName: \"kubernetes.io/projected/ed16245c-fd6f-458a-b6f5-07f82919b330-kube-api-access-d5rfd\") pod \"maas-api-revoke-keys-e2e-api-a-abc937-vv9ml\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.722226 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.722207 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc937-vv9ml\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.722685 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.722665 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc937-vv9ml\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.724568 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.724550 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abc937-vv9ml\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.775106 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.775071 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5rfd\" (UniqueName: \"kubernetes.io/projected/ed16245c-fd6f-458a-b6f5-07f82919b330-kube-api-access-d5rfd\") pod \"maas-api-revoke-keys-e2e-api-a-abc937-vv9ml\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.824290 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.824250 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:49.845858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.845830 5399 generic.go:356] "Generic (PLEG): container finished" podID="cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f" containerID="3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df" exitCode=0 Nov 02 20:28:49.845994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.845871 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" event={"ID":"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f","Type":"ContainerDied","Data":"3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df"} Nov 02 20:28:49.845994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.845889 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" Nov 02 20:28:49.845994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.845910 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl" event={"ID":"cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f","Type":"ContainerDied","Data":"fdc64f8353b317ff5f82df039fc3bdaa490c45f213165a0d8f30ebddc4924d96"} Nov 02 20:28:49.845994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.845931 5399 scope.go:122] "RemoveContainer" containerID="3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df" Nov 02 20:28:49.857424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.857403 5399 scope.go:122] "RemoveContainer" containerID="3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df" Nov 02 20:28:49.857787 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:28:49.857744 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df\": container with ID starting with 3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df not found: ID does not exist" containerID="3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df" Nov 02 20:28:49.857891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:49.857800 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df"} err="failed to get container status \"3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df\": rpc error: code = NotFound desc = could not find container \"3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df\": container with ID starting with 3eeb1f5ddb20d8f2bb28d6fc8950103d4b765c274f46f1d9fc721c86d7ab37df not found: ID does not exist" Nov 02 20:28:50.001826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:50.001784 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl"] Nov 02 20:28:50.005706 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:50.005678 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d5a51f-7786bb6f8c-67ccl"] Nov 02 20:28:50.198471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:50.198440 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml"] Nov 02 20:28:50.198694 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:28:50.198669 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded16245c_fd6f_458a_b6f5_07f82919b330.slice/crio-4c2f84bd8100ffeb72fd643f3f93f63a8e50b3f08a37529642112058c50756f3 WatchSource:0}: Error finding container 4c2f84bd8100ffeb72fd643f3f93f63a8e50b3f08a37529642112058c50756f3: Status 404 returned error can't find the container with id 4c2f84bd8100ffeb72fd643f3f93f63a8e50b3f08a37529642112058c50756f3 Nov 02 20:28:50.855065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:50.855023 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" event={"ID":"ed16245c-fd6f-458a-b6f5-07f82919b330","Type":"ContainerStarted","Data":"4c7f2480b4911dc7af2137964319458334056d2fc237d893e8c96cb1ea2b07d6"} Nov 02 20:28:50.855309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:50.855118 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" event={"ID":"ed16245c-fd6f-458a-b6f5-07f82919b330","Type":"ContainerStarted","Data":"4c2f84bd8100ffeb72fd643f3f93f63a8e50b3f08a37529642112058c50756f3"} Nov 02 20:28:50.963992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:50.963926 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" podStartSLOduration=1.9639112380000001 podStartE2EDuration="1.963911238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:50.960261781 +0000 UTC m=+2173.570913872" watchObservedRunningTime="2026-11-02 20:28:50.963911238 +0000 UTC m=+2173.574563378" Nov 02 20:28:51.511015 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:51.510939 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f" path="/var/lib/kubelet/pods/cd80cb29-3e01-48eb-9acf-7ca95bc5ad8f/volumes" Nov 02 20:28:51.865572 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:51.865532 5399 generic.go:356] "Generic (PLEG): container finished" podID="ed16245c-fd6f-458a-b6f5-07f82919b330" containerID="4c7f2480b4911dc7af2137964319458334056d2fc237d893e8c96cb1ea2b07d6" exitCode=0 Nov 02 20:28:51.865772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:51.865621 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" event={"ID":"ed16245c-fd6f-458a-b6f5-07f82919b330","Type":"ContainerDied","Data":"4c7f2480b4911dc7af2137964319458334056d2fc237d893e8c96cb1ea2b07d6"} Nov 02 20:28:53.042536 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.042510 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:53.075914 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.075883 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-openshift-ca\") pod \"ed16245c-fd6f-458a-b6f5-07f82919b330\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " Nov 02 20:28:53.076061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.075934 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-xks-ca\") pod \"ed16245c-fd6f-458a-b6f5-07f82919b330\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " Nov 02 20:28:53.076061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.076023 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed16245c-fd6f-458a-b6f5-07f82919b330-kube-api-access-d5rfd\" (UniqueName: \"kubernetes.io/projected/ed16245c-fd6f-458a-b6f5-07f82919b330-kube-api-access-d5rfd\") pod \"ed16245c-fd6f-458a-b6f5-07f82919b330\" (UID: \"ed16245c-fd6f-458a-b6f5-07f82919b330\") " Nov 02 20:28:53.076344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.076311 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-openshift-ca" pod "ed16245c-fd6f-458a-b6f5-07f82919b330" (UID: "ed16245c-fd6f-458a-b6f5-07f82919b330"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:53.076668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.076652 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:53.078049 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.078030 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed16245c-fd6f-458a-b6f5-07f82919b330-kube-api-access-d5rfd" pod "ed16245c-fd6f-458a-b6f5-07f82919b330" (UID: "ed16245c-fd6f-458a-b6f5-07f82919b330"). InnerVolumeSpecName "kube-api-access-d5rfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:53.078138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.078117 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-xks-ca" pod "ed16245c-fd6f-458a-b6f5-07f82919b330" (UID: "ed16245c-fd6f-458a-b6f5-07f82919b330"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:53.178399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.178304 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5rfd\" (UniqueName: \"kubernetes.io/projected/ed16245c-fd6f-458a-b6f5-07f82919b330-kube-api-access-d5rfd\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:53.178399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.178342 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed16245c-fd6f-458a-b6f5-07f82919b330-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:53.880777 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.880746 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" Nov 02 20:28:53.880947 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.880777 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml" event={"ID":"ed16245c-fd6f-458a-b6f5-07f82919b330","Type":"ContainerDied","Data":"4c2f84bd8100ffeb72fd643f3f93f63a8e50b3f08a37529642112058c50756f3"} Nov 02 20:28:53.880947 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:28:53.880829 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c2f84bd8100ffeb72fd643f3f93f63a8e50b3f08a37529642112058c50756f3" Nov 02 20:29:00.995036 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:00.994995 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl"] Nov 02 20:29:01.513219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.513189 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7"] Nov 02 20:29:01.513867 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.513851 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ed16245c-fd6f-458a-b6f5-07f82919b330" containerName="revoke-keys" Nov 02 20:29:01.513916 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.513867 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed16245c-fd6f-458a-b6f5-07f82919b330" containerName="revoke-keys" Nov 02 20:29:01.514018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.514008 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed16245c-fd6f-458a-b6f5-07f82919b330" containerName="revoke-keys" Nov 02 20:29:01.517761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.517743 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:01.556710 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.556677 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7"] Nov 02 20:29:01.566443 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.566408 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:29:01.566617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.566598 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:29:01.613165 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.613122 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-nqsj7\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:01.613435 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.613302 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8xl9\" (UniqueName: \"kubernetes.io/projected/468645d2-ef7c-401a-97bd-e7eeb894c964-kube-api-access-t8xl9\") pod \"maas-api-revoke-keys-e2e-mt-602be312-nqsj7\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:01.613508 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.613467 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-nqsj7\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:01.714813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.714777 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-nqsj7\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:01.714995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.714860 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8xl9\" (UniqueName: \"kubernetes.io/projected/468645d2-ef7c-401a-97bd-e7eeb894c964-kube-api-access-t8xl9\") pod \"maas-api-revoke-keys-e2e-mt-602be312-nqsj7\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:01.714995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.714943 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-nqsj7\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:01.715519 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.715490 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-nqsj7\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:01.717256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.717232 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-602be312-nqsj7\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:01.767767 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.767698 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8xl9\" (UniqueName: \"kubernetes.io/projected/468645d2-ef7c-401a-97bd-e7eeb894c964-kube-api-access-t8xl9\") pod \"maas-api-revoke-keys-e2e-mt-602be312-nqsj7\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:01.835956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:01.835920 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:02.211676 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:02.211644 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7"] Nov 02 20:29:02.212099 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:29:02.211992 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468645d2_ef7c_401a_97bd_e7eeb894c964.slice/crio-77b7a29c477b2fabe6141c953d35125cb809667c8cd6335705210348fde07be6 WatchSource:0}: Error finding container 77b7a29c477b2fabe6141c953d35125cb809667c8cd6335705210348fde07be6: Status 404 returned error can't find the container with id 77b7a29c477b2fabe6141c953d35125cb809667c8cd6335705210348fde07be6 Nov 02 20:29:02.956348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:02.955342 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" event={"ID":"468645d2-ef7c-401a-97bd-e7eeb894c964","Type":"ContainerStarted","Data":"13159b83946569ad8f18f7a0d01c40f63c591be828e76367de8c410f2fe38045"} Nov 02 20:29:02.956348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:02.955416 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" event={"ID":"468645d2-ef7c-401a-97bd-e7eeb894c964","Type":"ContainerStarted","Data":"77b7a29c477b2fabe6141c953d35125cb809667c8cd6335705210348fde07be6"} Nov 02 20:29:03.062184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:03.062128 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" podStartSLOduration=2.062109454 podStartE2EDuration="2.062109454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:03.062050043 +0000 UTC m=+2185.672702137" watchObservedRunningTime="2026-11-02 20:29:03.062109454 +0000 UTC m=+2185.672761545" Nov 02 20:29:03.966436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:03.966394 5399 generic.go:356] "Generic (PLEG): container finished" podID="468645d2-ef7c-401a-97bd-e7eeb894c964" containerID="13159b83946569ad8f18f7a0d01c40f63c591be828e76367de8c410f2fe38045" exitCode=0 Nov 02 20:29:03.966976 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:03.966484 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" event={"ID":"468645d2-ef7c-401a-97bd-e7eeb894c964","Type":"ContainerDied","Data":"13159b83946569ad8f18f7a0d01c40f63c591be828e76367de8c410f2fe38045"} Nov 02 20:29:05.144727 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.144698 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:05.199092 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.199058 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-openshift-ca\") pod \"468645d2-ef7c-401a-97bd-e7eeb894c964\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " Nov 02 20:29:05.199311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.199204 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-xks-ca\") pod \"468645d2-ef7c-401a-97bd-e7eeb894c964\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " Nov 02 20:29:05.199386 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.199326 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468645d2-ef7c-401a-97bd-e7eeb894c964-kube-api-access-t8xl9\" (UniqueName: \"kubernetes.io/projected/468645d2-ef7c-401a-97bd-e7eeb894c964-kube-api-access-t8xl9\") pod \"468645d2-ef7c-401a-97bd-e7eeb894c964\" (UID: \"468645d2-ef7c-401a-97bd-e7eeb894c964\") " Nov 02 20:29:05.199482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.199433 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-openshift-ca" pod "468645d2-ef7c-401a-97bd-e7eeb894c964" (UID: "468645d2-ef7c-401a-97bd-e7eeb894c964"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:05.200174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.200147 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:05.201541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.201514 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-xks-ca" pod "468645d2-ef7c-401a-97bd-e7eeb894c964" (UID: "468645d2-ef7c-401a-97bd-e7eeb894c964"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:05.201687 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.201634 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468645d2-ef7c-401a-97bd-e7eeb894c964-kube-api-access-t8xl9" pod "468645d2-ef7c-401a-97bd-e7eeb894c964" (UID: "468645d2-ef7c-401a-97bd-e7eeb894c964"). InnerVolumeSpecName "kube-api-access-t8xl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:05.302242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.302162 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468645d2-ef7c-401a-97bd-e7eeb894c964-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:05.302242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.302191 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8xl9\" (UniqueName: \"kubernetes.io/projected/468645d2-ef7c-401a-97bd-e7eeb894c964-kube-api-access-t8xl9\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:05.983704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.983664 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" Nov 02 20:29:05.983704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.983689 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7" event={"ID":"468645d2-ef7c-401a-97bd-e7eeb894c964","Type":"ContainerDied","Data":"77b7a29c477b2fabe6141c953d35125cb809667c8cd6335705210348fde07be6"} Nov 02 20:29:05.983936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.983742 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77b7a29c477b2fabe6141c953d35125cb809667c8cd6335705210348fde07be6" Nov 02 20:29:05.995347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:05.995288 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" podUID="9668e2fe-b325-44d5-8551-50e5f563a14e" containerName="payload-processing" containerID="cri-o://c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317" gracePeriod=25 Nov 02 20:29:06.296528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.296493 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:29:06.320587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.320557 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9668e2fe-b325-44d5-8551-50e5f563a14e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9668e2fe-b325-44d5-8551-50e5f563a14e-plugins-config-volume\") pod \"9668e2fe-b325-44d5-8551-50e5f563a14e\" (UID: \"9668e2fe-b325-44d5-8551-50e5f563a14e\") " Nov 02 20:29:06.320725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.320690 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9668e2fe-b325-44d5-8551-50e5f563a14e-kube-api-access-qjmxq\" (UniqueName: \"kubernetes.io/projected/9668e2fe-b325-44d5-8551-50e5f563a14e-kube-api-access-qjmxq\") pod \"9668e2fe-b325-44d5-8551-50e5f563a14e\" (UID: \"9668e2fe-b325-44d5-8551-50e5f563a14e\") " Nov 02 20:29:06.320999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.320973 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9668e2fe-b325-44d5-8551-50e5f563a14e-plugins-config-volume" pod "9668e2fe-b325-44d5-8551-50e5f563a14e" (UID: "9668e2fe-b325-44d5-8551-50e5f563a14e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:06.321381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.321365 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9668e2fe-b325-44d5-8551-50e5f563a14e-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:06.322794 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.322765 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9668e2fe-b325-44d5-8551-50e5f563a14e-kube-api-access-qjmxq" pod "9668e2fe-b325-44d5-8551-50e5f563a14e" (UID: "9668e2fe-b325-44d5-8551-50e5f563a14e"). InnerVolumeSpecName "kube-api-access-qjmxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:06.422797 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.422756 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjmxq\" (UniqueName: \"kubernetes.io/projected/9668e2fe-b325-44d5-8551-50e5f563a14e-kube-api-access-qjmxq\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:06.992781 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.992742 5399 generic.go:356] "Generic (PLEG): container finished" podID="9668e2fe-b325-44d5-8551-50e5f563a14e" containerID="c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317" exitCode=0 Nov 02 20:29:06.992991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.992790 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" Nov 02 20:29:06.992991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.992842 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" event={"ID":"9668e2fe-b325-44d5-8551-50e5f563a14e","Type":"ContainerDied","Data":"c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317"} Nov 02 20:29:06.992991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.992893 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl" event={"ID":"9668e2fe-b325-44d5-8551-50e5f563a14e","Type":"ContainerDied","Data":"9f28b0b942f7975aa1a556fc5946a84ec9c50e87538b15769016c2a2168ff768"} Nov 02 20:29:06.992991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:06.992915 5399 scope.go:122] "RemoveContainer" containerID="c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317" Nov 02 20:29:07.004874 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:07.004739 5399 scope.go:122] "RemoveContainer" containerID="c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317" Nov 02 20:29:07.005061 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:29:07.005041 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317\": container with ID starting with c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317 not found: ID does not exist" containerID="c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317" Nov 02 20:29:07.005097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:07.005069 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317"} err="failed to get container status \"c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317\": rpc error: code = NotFound desc = could not find container \"c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317\": container with ID starting with c4351fadb6ec212ca3be572b1d1ae31f478af59373e46fe3517b45e0d6269317 not found: ID does not exist" Nov 02 20:29:07.147050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:07.146998 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl"] Nov 02 20:29:07.192797 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:07.192767 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abc937-56d4cdc5cc-cc7gl"] Nov 02 20:29:07.509366 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:07.509335 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9668e2fe-b325-44d5-8551-50e5f563a14e" path="/var/lib/kubelet/pods/9668e2fe-b325-44d5-8551-50e5f563a14e/volumes" Nov 02 20:29:10.934424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:10.934394 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p"] Nov 02 20:29:10.935159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:10.935133 5399 cpu_manager.go:414] "containerMap: removing container" podUID="9668e2fe-b325-44d5-8551-50e5f563a14e" containerName="payload-processing" Nov 02 20:29:10.935235 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:10.935169 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9668e2fe-b325-44d5-8551-50e5f563a14e" containerName="payload-processing" Nov 02 20:29:10.935292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:10.935238 5399 cpu_manager.go:414] "containerMap: removing container" podUID="468645d2-ef7c-401a-97bd-e7eeb894c964" containerName="revoke-keys" Nov 02 20:29:10.935292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:10.935252 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468645d2-ef7c-401a-97bd-e7eeb894c964" containerName="revoke-keys" Nov 02 20:29:10.935499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:10.935484 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="468645d2-ef7c-401a-97bd-e7eeb894c964" containerName="revoke-keys" Nov 02 20:29:10.935535 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:10.935511 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9668e2fe-b325-44d5-8551-50e5f563a14e" containerName="payload-processing" Nov 02 20:29:10.941142 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:10.941123 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:10.990204 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:10.990171 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1ab501-openshift-default-dockercfg-664zg\"" Nov 02 20:29:11.024455 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.024417 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxvj\" (UniqueName: \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-kube-api-access-9zxvj\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.024455 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.024455 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32778100-8a2b-46c9-b507-2273c72e10af-istiod-ca-cert\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.024658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.024632 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-envoy\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.024777 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.024756 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-istio-token\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.024837 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.024823 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-credential-socket\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.024898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.024882 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-data\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.024931 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.024913 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-certs\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.025014 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.024997 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-socket\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.025066 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.025051 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32778100-8a2b-46c9-b507-2273c72e10af-istio-podinfo\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.041175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.041146 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p"] Nov 02 20:29:11.126664 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.126629 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxvj\" (UniqueName: \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-kube-api-access-9zxvj\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.126664 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.126670 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32778100-8a2b-46c9-b507-2273c72e10af-istiod-ca-cert\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.126929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.126731 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-envoy\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.126929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.126809 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-istio-token\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.126929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.126834 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-credential-socket\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.126929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.126903 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-data\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.127325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.126934 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-certs\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.127325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.127009 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-socket\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.127325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.127055 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32778100-8a2b-46c9-b507-2273c72e10af-istio-podinfo\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.127621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.127557 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-credential-socket\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.127621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.127589 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-certs\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.127844 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.127556 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32778100-8a2b-46c9-b507-2273c72e10af-istiod-ca-cert\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.127844 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.127756 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-socket\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.127972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.127850 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-data\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.129959 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.129939 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-envoy\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.130397 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.130372 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32778100-8a2b-46c9-b507-2273c72e10af-istio-podinfo\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.182658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.182621 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxvj\" (UniqueName: \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-kube-api-access-9zxvj\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.183699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.183675 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-istio-token\") pod \"e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.257072 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.256985 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:11.706814 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:29:11.706782 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32778100_8a2b_46c9_b507_2273c72e10af.slice/crio-15139c6641c27ef705d7f3762dec015abd94866bb56e4de79ec5fc37ed8130b5 WatchSource:0}: Error finding container 15139c6641c27ef705d7f3762dec015abd94866bb56e4de79ec5fc37ed8130b5: Status 404 returned error can't find the container with id 15139c6641c27ef705d7f3762dec015abd94866bb56e4de79ec5fc37ed8130b5 Nov 02 20:29:11.706933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.706874 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p"] Nov 02 20:29:11.709732 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.709700 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:29:11.709818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.709763 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:29:11.709818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:11.709789 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:29:12.035660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:12.035570 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" event={"ID":"32778100-8a2b-46c9-b507-2273c72e10af","Type":"ContainerStarted","Data":"699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57"} Nov 02 20:29:12.035660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:12.035624 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" event={"ID":"32778100-8a2b-46c9-b507-2273c72e10af","Type":"ContainerStarted","Data":"15139c6641c27ef705d7f3762dec015abd94866bb56e4de79ec5fc37ed8130b5"} Nov 02 20:29:12.150525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:12.150469 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" podStartSLOduration=2.150453321 podStartE2EDuration="2.150453321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:12.145946082 +0000 UTC m=+2194.756598186" watchObservedRunningTime="2026-11-02 20:29:12.150453321 +0000 UTC m=+2194.761105414" Nov 02 20:29:12.266601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:12.257715 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:12.276728 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:12.276685 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:13.050057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:13.050016 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:13.052055 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:13.052030 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:29:14.207703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.207672 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl"] Nov 02 20:29:14.208124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.207864 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" podUID="76601a34-8e82-40c5-bc27-7960292e3183" containerName="maas-api" containerID="cri-o://d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c" gracePeriod=30 Nov 02 20:29:14.544920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.544732 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:29:14.709383 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.709346 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76601a34-8e82-40c5-bc27-7960292e3183-kube-api-access-tw8vk\" (UniqueName: \"kubernetes.io/projected/76601a34-8e82-40c5-bc27-7960292e3183-kube-api-access-tw8vk\") pod \"76601a34-8e82-40c5-bc27-7960292e3183\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " Nov 02 20:29:14.709563 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.709400 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-maas-api-tls\") pod \"76601a34-8e82-40c5-bc27-7960292e3183\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " Nov 02 20:29:14.709563 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.709537 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76601a34-8e82-40c5-bc27-7960292e3183-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76601a34-8e82-40c5-bc27-7960292e3183-openshift-service-ca\") pod \"76601a34-8e82-40c5-bc27-7960292e3183\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " Nov 02 20:29:14.709667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.709657 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-metrics-certs\") pod \"76601a34-8e82-40c5-bc27-7960292e3183\" (UID: \"76601a34-8e82-40c5-bc27-7960292e3183\") " Nov 02 20:29:14.709929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.709901 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76601a34-8e82-40c5-bc27-7960292e3183-openshift-service-ca" pod "76601a34-8e82-40c5-bc27-7960292e3183" (UID: "76601a34-8e82-40c5-bc27-7960292e3183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:14.710399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.710375 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76601a34-8e82-40c5-bc27-7960292e3183-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:14.712161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.712134 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-metrics-certs" pod "76601a34-8e82-40c5-bc27-7960292e3183" (UID: "76601a34-8e82-40c5-bc27-7960292e3183"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:14.712688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.712662 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-maas-api-tls" pod "76601a34-8e82-40c5-bc27-7960292e3183" (UID: "76601a34-8e82-40c5-bc27-7960292e3183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:14.712814 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.712712 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76601a34-8e82-40c5-bc27-7960292e3183-kube-api-access-tw8vk" pod "76601a34-8e82-40c5-bc27-7960292e3183" (UID: "76601a34-8e82-40c5-bc27-7960292e3183"). InnerVolumeSpecName "kube-api-access-tw8vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:14.811854 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.811804 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:14.811854 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.811838 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw8vk\" (UniqueName: \"kubernetes.io/projected/76601a34-8e82-40c5-bc27-7960292e3183-kube-api-access-tw8vk\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:14.811854 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:14.811850 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76601a34-8e82-40c5-bc27-7960292e3183-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:15.066756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:15.066666 5399 generic.go:356] "Generic (PLEG): container finished" podID="76601a34-8e82-40c5-bc27-7960292e3183" containerID="d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c" exitCode=0 Nov 02 20:29:15.066756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:15.066721 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" Nov 02 20:29:15.066963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:15.066749 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" event={"ID":"76601a34-8e82-40c5-bc27-7960292e3183","Type":"ContainerDied","Data":"d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c"} Nov 02 20:29:15.066963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:15.066810 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl" event={"ID":"76601a34-8e82-40c5-bc27-7960292e3183","Type":"ContainerDied","Data":"4e9f4b478139990ee3d42f344c8539003abfeef5109d4e9778b966c56ce33fc5"} Nov 02 20:29:15.066963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:15.066840 5399 scope.go:122] "RemoveContainer" containerID="d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c" Nov 02 20:29:15.079764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:15.079742 5399 scope.go:122] "RemoveContainer" containerID="d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c" Nov 02 20:29:15.080211 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:29:15.080181 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c\": container with ID starting with d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c not found: ID does not exist" containerID="d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c" Nov 02 20:29:15.080347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:15.080222 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c"} err="failed to get container status \"d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c\": rpc error: code = NotFound desc = could not find container \"d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c\": container with ID starting with d3a80ee3d66c7b764ac3c29f11912ada02781afedae18750ae08a3c6f33a447c not found: ID does not exist" Nov 02 20:29:15.226511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:15.226481 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl"] Nov 02 20:29:15.232366 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:15.232336 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abc937-59db9486d4-rzwtl"] Nov 02 20:29:15.509491 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:15.509411 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76601a34-8e82-40c5-bc27-7960292e3183" path="/var/lib/kubelet/pods/76601a34-8e82-40c5-bc27-7960292e3183/volumes" Nov 02 20:29:16.413425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:16.413389 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml"] Nov 02 20:29:16.459361 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:16.459327 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abc937-vv9ml"] Nov 02 20:29:17.511099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:17.511068 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed16245c-fd6f-458a-b6f5-07f82919b330" path="/var/lib/kubelet/pods/ed16245c-fd6f-458a-b6f5-07f82919b330/volumes" Nov 02 20:29:17.709017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:17.708982 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm"] Nov 02 20:29:17.709203 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:17.709182 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" podUID="de626de0-9f65-47f7-a705-f31a45b24f86" containerName="istio-proxy" containerID="cri-o://01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59" gracePeriod=30 Nov 02 20:29:18.238807 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:29:18.238745 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:29:20.338834 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:20.338795 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d"] Nov 02 20:29:20.339298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:20.339037 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" podUID="6b27d65e-440e-41df-b238-c9cbdd902ed2" containerName="istio-proxy" containerID="cri-o://da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914" gracePeriod=30 Nov 02 20:29:23.024716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.024685 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:29:23.229303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.226582 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-credential-socket\") pod \"de626de0-9f65-47f7-a705-f31a45b24f86\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " Nov 02 20:29:23.229303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.226736 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-certs\") pod \"de626de0-9f65-47f7-a705-f31a45b24f86\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " Nov 02 20:29:23.229303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.226800 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-data\") pod \"de626de0-9f65-47f7-a705-f31a45b24f86\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " Nov 02 20:29:23.229303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.226888 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-socket\") pod \"de626de0-9f65-47f7-a705-f31a45b24f86\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " Nov 02 20:29:23.229303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.226924 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-envoy\") pod \"de626de0-9f65-47f7-a705-f31a45b24f86\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " Nov 02 20:29:23.229303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.226996 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-istio-token\" (UniqueName: \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-istio-token\") pod \"de626de0-9f65-47f7-a705-f31a45b24f86\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " Nov 02 20:29:23.229303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.227123 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de626de0-9f65-47f7-a705-f31a45b24f86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de626de0-9f65-47f7-a705-f31a45b24f86-istio-podinfo\") pod \"de626de0-9f65-47f7-a705-f31a45b24f86\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " Nov 02 20:29:23.229303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.227195 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-kube-api-access-md6x2\" (UniqueName: \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-kube-api-access-md6x2\") pod \"de626de0-9f65-47f7-a705-f31a45b24f86\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " Nov 02 20:29:23.229303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.227241 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de626de0-9f65-47f7-a705-f31a45b24f86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de626de0-9f65-47f7-a705-f31a45b24f86-istiod-ca-cert\") pod \"de626de0-9f65-47f7-a705-f31a45b24f86\" (UID: \"de626de0-9f65-47f7-a705-f31a45b24f86\") " Nov 02 20:29:23.229303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.229061 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de626de0-9f65-47f7-a705-f31a45b24f86-istiod-ca-cert" pod "de626de0-9f65-47f7-a705-f31a45b24f86" (UID: "de626de0-9f65-47f7-a705-f31a45b24f86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:23.229815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.229438 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-credential-socket" pod "de626de0-9f65-47f7-a705-f31a45b24f86" (UID: "de626de0-9f65-47f7-a705-f31a45b24f86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:23.229815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.229656 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-certs" pod "de626de0-9f65-47f7-a705-f31a45b24f86" (UID: "de626de0-9f65-47f7-a705-f31a45b24f86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:23.229891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.229840 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-data" pod "de626de0-9f65-47f7-a705-f31a45b24f86" (UID: "de626de0-9f65-47f7-a705-f31a45b24f86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:23.233663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.233254 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-socket" pod "de626de0-9f65-47f7-a705-f31a45b24f86" (UID: "de626de0-9f65-47f7-a705-f31a45b24f86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:23.238391 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.238359 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-istio-token" pod "de626de0-9f65-47f7-a705-f31a45b24f86" (UID: "de626de0-9f65-47f7-a705-f31a45b24f86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:23.245536 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.243145 5399 generic.go:356] "Generic (PLEG): container finished" podID="de626de0-9f65-47f7-a705-f31a45b24f86" containerID="01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59" exitCode=0 Nov 02 20:29:23.246037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.243185 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" event={"ID":"de626de0-9f65-47f7-a705-f31a45b24f86","Type":"ContainerDied","Data":"01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59"} Nov 02 20:29:23.246246 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.246224 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" event={"ID":"de626de0-9f65-47f7-a705-f31a45b24f86","Type":"ContainerDied","Data":"8d06cccef49f0b7dbd08e53c8ba71148320b8cb505d3865743197a229d62d8ba"} Nov 02 20:29:23.246424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.246408 5399 scope.go:122] "RemoveContainer" containerID="01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59" Nov 02 20:29:23.246673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.243299 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm" Nov 02 20:29:23.256774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.256659 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de626de0-9f65-47f7-a705-f31a45b24f86-istio-podinfo" pod "de626de0-9f65-47f7-a705-f31a45b24f86" (UID: "de626de0-9f65-47f7-a705-f31a45b24f86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:29:23.257018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.256994 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-kube-api-access-md6x2" pod "de626de0-9f65-47f7-a705-f31a45b24f86" (UID: "de626de0-9f65-47f7-a705-f31a45b24f86"). InnerVolumeSpecName "kube-api-access-md6x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:23.257440 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.257395 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-envoy" pod "de626de0-9f65-47f7-a705-f31a45b24f86" (UID: "de626de0-9f65-47f7-a705-f31a45b24f86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:23.287144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.287122 5399 scope.go:122] "RemoveContainer" containerID="01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59" Nov 02 20:29:23.287576 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:29:23.287554 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59\": container with ID starting with 01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59 not found: ID does not exist" containerID="01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59" Nov 02 20:29:23.287669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.287585 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59"} err="failed to get container status \"01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59\": rpc error: code = NotFound desc = could not find container \"01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59\": container with ID starting with 01717a3a1b086469193ee52a7d8536872c46e392793abe70e8bc82b11fd2ea59 not found: ID does not exist" Nov 02 20:29:23.330194 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.330156 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:23.330194 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.330193 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:23.330528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.330211 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:23.330528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.330224 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:23.330528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.330236 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:23.330528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.330296 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de626de0-9f65-47f7-a705-f31a45b24f86-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:23.330528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.330312 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md6x2\" (UniqueName: \"kubernetes.io/projected/de626de0-9f65-47f7-a705-f31a45b24f86-kube-api-access-md6x2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:23.330528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.330327 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de626de0-9f65-47f7-a705-f31a45b24f86-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:23.330528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.330339 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de626de0-9f65-47f7-a705-f31a45b24f86-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:23.656339 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.656307 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm"] Nov 02 20:29:23.664683 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:23.664659 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-zslvm"] Nov 02 20:29:25.046162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.046071 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f"] Nov 02 20:29:25.091755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.091723 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b62629a1-qlz5f"] Nov 02 20:29:25.510444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.510409 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf408fe6-a44d-479c-9aa6-a9ce068124c5" path="/var/lib/kubelet/pods/cf408fe6-a44d-479c-9aa6-a9ce068124c5/volumes" Nov 02 20:29:25.511012 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.510994 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de626de0-9f65-47f7-a705-f31a45b24f86" path="/var/lib/kubelet/pods/de626de0-9f65-47f7-a705-f31a45b24f86/volumes" Nov 02 20:29:25.629913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.629883 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:29:25.663837 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.663809 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-kube-api-access-mfhnk\" (UniqueName: \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-kube-api-access-mfhnk\") pod \"6b27d65e-440e-41df-b238-c9cbdd902ed2\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " Nov 02 20:29:25.663987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.663847 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b27d65e-440e-41df-b238-c9cbdd902ed2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b27d65e-440e-41df-b238-c9cbdd902ed2-istiod-ca-cert\") pod \"6b27d65e-440e-41df-b238-c9cbdd902ed2\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " Nov 02 20:29:25.663987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.663886 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-envoy\") pod \"6b27d65e-440e-41df-b238-c9cbdd902ed2\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " Nov 02 20:29:25.663987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.663926 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-certs\") pod \"6b27d65e-440e-41df-b238-c9cbdd902ed2\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " Nov 02 20:29:25.663987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.663979 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-credential-socket\") pod \"6b27d65e-440e-41df-b238-c9cbdd902ed2\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " Nov 02 20:29:25.664220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.664041 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-socket\") pod \"6b27d65e-440e-41df-b238-c9cbdd902ed2\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " Nov 02 20:29:25.664220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.664059 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-podinfo\") pod \"6b27d65e-440e-41df-b238-c9cbdd902ed2\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " Nov 02 20:29:25.664220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.664098 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-data\") pod \"6b27d65e-440e-41df-b238-c9cbdd902ed2\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " Nov 02 20:29:25.664428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.664360 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-token\" (UniqueName: \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-token\") pod \"6b27d65e-440e-41df-b238-c9cbdd902ed2\" (UID: \"6b27d65e-440e-41df-b238-c9cbdd902ed2\") " Nov 02 20:29:25.664428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.664384 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-certs" pod "6b27d65e-440e-41df-b238-c9cbdd902ed2" (UID: "6b27d65e-440e-41df-b238-c9cbdd902ed2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:25.664544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.664424 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b27d65e-440e-41df-b238-c9cbdd902ed2-istiod-ca-cert" pod "6b27d65e-440e-41df-b238-c9cbdd902ed2" (UID: "6b27d65e-440e-41df-b238-c9cbdd902ed2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:25.664544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.664507 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-credential-socket" pod "6b27d65e-440e-41df-b238-c9cbdd902ed2" (UID: "6b27d65e-440e-41df-b238-c9cbdd902ed2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:25.664647 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.664625 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-data" pod "6b27d65e-440e-41df-b238-c9cbdd902ed2" (UID: "6b27d65e-440e-41df-b238-c9cbdd902ed2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:25.664900 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.664876 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-socket" pod "6b27d65e-440e-41df-b238-c9cbdd902ed2" (UID: "6b27d65e-440e-41df-b238-c9cbdd902ed2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:25.665100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.665082 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.665170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.665102 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.665170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.665111 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b27d65e-440e-41df-b238-c9cbdd902ed2-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.665170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.665121 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.665170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.665129 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.666302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.666243 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-kube-api-access-mfhnk" pod "6b27d65e-440e-41df-b238-c9cbdd902ed2" (UID: "6b27d65e-440e-41df-b238-c9cbdd902ed2"). InnerVolumeSpecName "kube-api-access-mfhnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:25.666549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.666527 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-envoy" pod "6b27d65e-440e-41df-b238-c9cbdd902ed2" (UID: "6b27d65e-440e-41df-b238-c9cbdd902ed2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:25.666624 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.666579 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-podinfo" pod "6b27d65e-440e-41df-b238-c9cbdd902ed2" (UID: "6b27d65e-440e-41df-b238-c9cbdd902ed2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:29:25.666702 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.666682 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-token" pod "6b27d65e-440e-41df-b238-c9cbdd902ed2" (UID: "6b27d65e-440e-41df-b238-c9cbdd902ed2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:25.766518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.766480 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.766518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.766513 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfhnk\" (UniqueName: \"kubernetes.io/projected/6b27d65e-440e-41df-b238-c9cbdd902ed2-kube-api-access-mfhnk\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.766518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.766525 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:25.766827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:25.766533 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b27d65e-440e-41df-b238-c9cbdd902ed2-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:26.272144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:26.272105 5399 generic.go:356] "Generic (PLEG): container finished" podID="6b27d65e-440e-41df-b238-c9cbdd902ed2" containerID="da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914" exitCode=0 Nov 02 20:29:26.272743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:26.272178 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" event={"ID":"6b27d65e-440e-41df-b238-c9cbdd902ed2","Type":"ContainerDied","Data":"da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914"} Nov 02 20:29:26.272743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:26.272165 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" Nov 02 20:29:26.272743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:26.272247 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d" event={"ID":"6b27d65e-440e-41df-b238-c9cbdd902ed2","Type":"ContainerDied","Data":"89822f500178474c403aea29d5c174fffe0e31b5cdf26f35da3e09c4001d9e0d"} Nov 02 20:29:26.272743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:26.272318 5399 scope.go:122] "RemoveContainer" containerID="da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914" Nov 02 20:29:26.284940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:26.284920 5399 scope.go:122] "RemoveContainer" containerID="da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914" Nov 02 20:29:26.285192 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:29:26.285171 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914\": container with ID starting with da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914 not found: ID does not exist" containerID="da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914" Nov 02 20:29:26.285231 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:26.285204 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914"} err="failed to get container status \"da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914\": rpc error: code = NotFound desc = could not find container \"da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914\": container with ID starting with da028fdd94f4c18d5e25311b2e1253789ab9fa15a8647e8d33ea9a85691b8914 not found: ID does not exist" Nov 02 20:29:26.473401 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:26.473361 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d"] Nov 02 20:29:26.525244 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:26.525139 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-b9d6d"] Nov 02 20:29:27.522293 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:27.522239 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b27d65e-440e-41df-b238-c9cbdd902ed2" path="/var/lib/kubelet/pods/6b27d65e-440e-41df-b238-c9cbdd902ed2/volumes" Nov 02 20:29:35.394613 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.394576 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck"] Nov 02 20:29:35.395146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.394775 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" podUID="dbb75c5d-da33-49c5-aeda-03a3e0e74a5f" containerName="maas-api" containerID="cri-o://2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b" gracePeriod=30 Nov 02 20:29:35.693368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.693335 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:29:35.816053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.814603 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-maas-api-tls\") pod \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " Nov 02 20:29:35.816053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.814708 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-openshift-service-ca\") pod \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " Nov 02 20:29:35.816053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.814811 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-metrics-certs\") pod \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " Nov 02 20:29:35.816053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.815047 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-kube-api-access-f46tg\" (UniqueName: \"kubernetes.io/projected/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-kube-api-access-f46tg\") pod \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\" (UID: \"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f\") " Nov 02 20:29:35.816488 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.816230 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-openshift-service-ca" pod "dbb75c5d-da33-49c5-aeda-03a3e0e74a5f" (UID: "dbb75c5d-da33-49c5-aeda-03a3e0e74a5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:35.818381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.818338 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-kube-api-access-f46tg" pod "dbb75c5d-da33-49c5-aeda-03a3e0e74a5f" (UID: "dbb75c5d-da33-49c5-aeda-03a3e0e74a5f"). InnerVolumeSpecName "kube-api-access-f46tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:35.818972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.818943 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-metrics-certs" pod "dbb75c5d-da33-49c5-aeda-03a3e0e74a5f" (UID: "dbb75c5d-da33-49c5-aeda-03a3e0e74a5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:35.819123 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.819099 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-maas-api-tls" pod "dbb75c5d-da33-49c5-aeda-03a3e0e74a5f" (UID: "dbb75c5d-da33-49c5-aeda-03a3e0e74a5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:35.918466 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.918370 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:35.918466 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.918412 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f46tg\" (UniqueName: \"kubernetes.io/projected/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-kube-api-access-f46tg\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:35.918466 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.918428 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:35.918466 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:35.918444 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:36.373870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:36.373829 5399 generic.go:356] "Generic (PLEG): container finished" podID="dbb75c5d-da33-49c5-aeda-03a3e0e74a5f" containerID="2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b" exitCode=0 Nov 02 20:29:36.374338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:36.374168 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" Nov 02 20:29:36.395819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:36.374191 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" event={"ID":"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f","Type":"ContainerDied","Data":"2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b"} Nov 02 20:29:36.395819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:36.395185 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck" event={"ID":"dbb75c5d-da33-49c5-aeda-03a3e0e74a5f","Type":"ContainerDied","Data":"7301a84057c8d933fe6b7b32d763e977506b96539157d05ddb236e946991d616"} Nov 02 20:29:36.395819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:36.395227 5399 scope.go:122] "RemoveContainer" containerID="2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b" Nov 02 20:29:36.527624 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:36.527586 5399 scope.go:122] "RemoveContainer" containerID="2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b" Nov 02 20:29:36.534838 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:29:36.529815 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b\": container with ID starting with 2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b not found: ID does not exist" containerID="2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b" Nov 02 20:29:36.534838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:36.529857 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b"} err="failed to get container status \"2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b\": rpc error: code = NotFound desc = could not find container \"2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b\": container with ID starting with 2a59b9f8456f891ef47674cf70a430109f1c68931a714de50e158e804a05161b not found: ID does not exist" Nov 02 20:29:36.583315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:36.583285 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck"] Nov 02 20:29:36.629460 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:36.629384 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dcb03-5cc774dbf9-8sxck"] Nov 02 20:29:37.509530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:37.509496 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb75c5d-da33-49c5-aeda-03a3e0e74a5f" path="/var/lib/kubelet/pods/dbb75c5d-da33-49c5-aeda-03a3e0e74a5f/volumes" Nov 02 20:29:38.465484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:38.465453 5399 scope.go:122] "RemoveContainer" containerID="9adce58c47efecb7c2d0404e751876a622beb59e5025e0804a215fec340f8cf2" Nov 02 20:29:41.411842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.411808 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn"] Nov 02 20:29:41.412309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.412000 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" podUID="e92863ab-b872-43ad-a152-709196731a86" containerName="maas-api" containerID="cri-o://0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0" gracePeriod=30 Nov 02 20:29:41.750117 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.750091 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:29:41.824629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.824593 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-metrics-certs\") pod \"e92863ab-b872-43ad-a152-709196731a86\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " Nov 02 20:29:41.824835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.824818 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-maas-api-tls\") pod \"e92863ab-b872-43ad-a152-709196731a86\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " Nov 02 20:29:41.824919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.824902 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92863ab-b872-43ad-a152-709196731a86-kube-api-access-8m6dj\" (UniqueName: \"kubernetes.io/projected/e92863ab-b872-43ad-a152-709196731a86-kube-api-access-8m6dj\") pod \"e92863ab-b872-43ad-a152-709196731a86\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " Nov 02 20:29:41.825012 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.824997 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92863ab-b872-43ad-a152-709196731a86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92863ab-b872-43ad-a152-709196731a86-openshift-service-ca\") pod \"e92863ab-b872-43ad-a152-709196731a86\" (UID: \"e92863ab-b872-43ad-a152-709196731a86\") " Nov 02 20:29:41.826545 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.826490 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92863ab-b872-43ad-a152-709196731a86-openshift-service-ca" pod "e92863ab-b872-43ad-a152-709196731a86" (UID: "e92863ab-b872-43ad-a152-709196731a86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:41.828003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.827961 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92863ab-b872-43ad-a152-709196731a86-kube-api-access-8m6dj" pod "e92863ab-b872-43ad-a152-709196731a86" (UID: "e92863ab-b872-43ad-a152-709196731a86"). InnerVolumeSpecName "kube-api-access-8m6dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:41.828779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.828749 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-maas-api-tls" pod "e92863ab-b872-43ad-a152-709196731a86" (UID: "e92863ab-b872-43ad-a152-709196731a86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:41.830086 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.830049 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-metrics-certs" pod "e92863ab-b872-43ad-a152-709196731a86" (UID: "e92863ab-b872-43ad-a152-709196731a86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:41.932345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.927649 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92863ab-b872-43ad-a152-709196731a86-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:41.932345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.927704 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:41.932345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.927718 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92863ab-b872-43ad-a152-709196731a86-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:41.932345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:41.927733 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6dj\" (UniqueName: \"kubernetes.io/projected/e92863ab-b872-43ad-a152-709196731a86-kube-api-access-8m6dj\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:42.486471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:42.482226 5399 generic.go:356] "Generic (PLEG): container finished" podID="e92863ab-b872-43ad-a152-709196731a86" containerID="0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0" exitCode=0 Nov 02 20:29:42.486471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:42.482403 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" event={"ID":"e92863ab-b872-43ad-a152-709196731a86","Type":"ContainerDied","Data":"0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0"} Nov 02 20:29:42.486471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:42.482460 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" event={"ID":"e92863ab-b872-43ad-a152-709196731a86","Type":"ContainerDied","Data":"69b019d8b779eecce1731b5dcd77d6c4081e9373d863f398598c424445977c47"} Nov 02 20:29:42.486471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:42.482496 5399 scope.go:122] "RemoveContainer" containerID="0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0" Nov 02 20:29:42.486471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:42.482660 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn" Nov 02 20:29:42.581539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:42.581511 5399 scope.go:122] "RemoveContainer" containerID="0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0" Nov 02 20:29:42.581962 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:29:42.581934 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0\": container with ID starting with 0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0 not found: ID does not exist" containerID="0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0" Nov 02 20:29:42.582049 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:42.581976 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0"} err="failed to get container status \"0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0\": rpc error: code = NotFound desc = could not find container \"0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0\": container with ID starting with 0d84262d84a2a065f0840948789af705409d736cae81ee4896d8a7d229461ba0 not found: ID does not exist" Nov 02 20:29:42.674224 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:42.674187 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn"] Nov 02 20:29:42.720704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:42.720671 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-81e58f-7c7cc7df7d-wbpxn"] Nov 02 20:29:43.525025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.524986 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92863ab-b872-43ad-a152-709196731a86" path="/var/lib/kubelet/pods/e92863ab-b872-43ad-a152-709196731a86/volumes" Nov 02 20:29:43.552792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.552758 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb"] Nov 02 20:29:43.553419 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553405 5399 cpu_manager.go:414] "containerMap: removing container" podUID="dbb75c5d-da33-49c5-aeda-03a3e0e74a5f" containerName="maas-api" Nov 02 20:29:43.553464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553421 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb75c5d-da33-49c5-aeda-03a3e0e74a5f" containerName="maas-api" Nov 02 20:29:43.553464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553448 5399 cpu_manager.go:414] "containerMap: removing container" podUID="76601a34-8e82-40c5-bc27-7960292e3183" containerName="maas-api" Nov 02 20:29:43.553464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553453 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76601a34-8e82-40c5-bc27-7960292e3183" containerName="maas-api" Nov 02 20:29:43.553575 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553466 5399 cpu_manager.go:414] "containerMap: removing container" podUID="de626de0-9f65-47f7-a705-f31a45b24f86" containerName="istio-proxy" Nov 02 20:29:43.553575 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553472 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de626de0-9f65-47f7-a705-f31a45b24f86" containerName="istio-proxy" Nov 02 20:29:43.553575 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553485 5399 cpu_manager.go:414] "containerMap: removing container" podUID="6b27d65e-440e-41df-b238-c9cbdd902ed2" containerName="istio-proxy" Nov 02 20:29:43.553575 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553490 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b27d65e-440e-41df-b238-c9cbdd902ed2" containerName="istio-proxy" Nov 02 20:29:43.553575 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553506 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e92863ab-b872-43ad-a152-709196731a86" containerName="maas-api" Nov 02 20:29:43.553575 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553511 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92863ab-b872-43ad-a152-709196731a86" containerName="maas-api" Nov 02 20:29:43.553750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553601 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb75c5d-da33-49c5-aeda-03a3e0e74a5f" containerName="maas-api" Nov 02 20:29:43.553750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553614 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="76601a34-8e82-40c5-bc27-7960292e3183" containerName="maas-api" Nov 02 20:29:43.553750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553621 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="de626de0-9f65-47f7-a705-f31a45b24f86" containerName="istio-proxy" Nov 02 20:29:43.553750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553629 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b27d65e-440e-41df-b238-c9cbdd902ed2" containerName="istio-proxy" Nov 02 20:29:43.553750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.553636 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92863ab-b872-43ad-a152-709196731a86" containerName="maas-api" Nov 02 20:29:43.558704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.558681 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:29:43.568695 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.568662 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zzp9\" (UniqueName: \"kubernetes.io/projected/e7dc89db-90d6-4715-8b34-062ac0779e15-kube-api-access-5zzp9\") pod \"payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb\" (UID: \"e7dc89db-90d6-4715-8b34-062ac0779e15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:29:43.568846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.568822 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7dc89db-90d6-4715-8b34-062ac0779e15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb\" (UID: \"e7dc89db-90d6-4715-8b34-062ac0779e15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:29:43.616592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.616558 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-62d287-dockercfg-gqc84\"" Nov 02 20:29:43.616592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.616555 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-62d287\"" Nov 02 20:29:43.617064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.617043 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb"] Nov 02 20:29:43.729305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.676440 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzp9\" (UniqueName: \"kubernetes.io/projected/e7dc89db-90d6-4715-8b34-062ac0779e15-kube-api-access-5zzp9\") pod \"payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb\" (UID: \"e7dc89db-90d6-4715-8b34-062ac0779e15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:29:43.729305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.676626 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7dc89db-90d6-4715-8b34-062ac0779e15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb\" (UID: \"e7dc89db-90d6-4715-8b34-062ac0779e15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:29:43.729305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.678114 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7dc89db-90d6-4715-8b34-062ac0779e15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb\" (UID: \"e7dc89db-90d6-4715-8b34-062ac0779e15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:29:43.814349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.806754 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzp9\" (UniqueName: \"kubernetes.io/projected/e7dc89db-90d6-4715-8b34-062ac0779e15-kube-api-access-5zzp9\") pod \"payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb\" (UID: \"e7dc89db-90d6-4715-8b34-062ac0779e15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:29:43.838300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.837355 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5"] Nov 02 20:29:43.845847 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.845812 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5"] Nov 02 20:29:43.846017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.845947 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:29:43.872623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.872595 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:29:43.983965 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.983387 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnsvc\" (UniqueName: \"kubernetes.io/projected/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-kube-api-access-wnsvc\") pod \"payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5\" (UID: \"67e804bb-46a1-4963-af5d-f0fd6a80bdbc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:29:43.983965 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:43.983468 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5\" (UID: \"67e804bb-46a1-4963-af5d-f0fd6a80bdbc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:29:44.085799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.085705 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnsvc\" (UniqueName: \"kubernetes.io/projected/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-kube-api-access-wnsvc\") pod \"payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5\" (UID: \"67e804bb-46a1-4963-af5d-f0fd6a80bdbc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:29:44.085799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.085793 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5\" (UID: \"67e804bb-46a1-4963-af5d-f0fd6a80bdbc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:29:44.086546 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.086524 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5\" (UID: \"67e804bb-46a1-4963-af5d-f0fd6a80bdbc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:29:44.138987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.138954 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnsvc\" (UniqueName: \"kubernetes.io/projected/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-kube-api-access-wnsvc\") pod \"payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5\" (UID: \"67e804bb-46a1-4963-af5d-f0fd6a80bdbc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:29:44.175045 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.175013 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:29:44.304358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.301629 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb"] Nov 02 20:29:44.308351 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:29:44.305669 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dc89db_90d6_4715_8b34_062ac0779e15.slice/crio-4cc48aa19561708fa0c432ecc74e5f7a49360eecb9a19cbf4c90cdfe55966db8 WatchSource:0}: Error finding container 4cc48aa19561708fa0c432ecc74e5f7a49360eecb9a19cbf4c90cdfe55966db8: Status 404 returned error can't find the container with id 4cc48aa19561708fa0c432ecc74e5f7a49360eecb9a19cbf4c90cdfe55966db8 Nov 02 20:29:44.507176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.507137 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" event={"ID":"e7dc89db-90d6-4715-8b34-062ac0779e15","Type":"ContainerStarted","Data":"4f0625dd6d8ec1dd275c37ce71c5b936aee4c9b6eb21e4688a26001ebacf3cff"} Nov 02 20:29:44.507377 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.507187 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" event={"ID":"e7dc89db-90d6-4715-8b34-062ac0779e15","Type":"ContainerStarted","Data":"4cc48aa19561708fa0c432ecc74e5f7a49360eecb9a19cbf4c90cdfe55966db8"} Nov 02 20:29:44.507377 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.507264 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:29:44.550457 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.550423 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5"] Nov 02 20:29:44.550989 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:29:44.550960 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e804bb_46a1_4963_af5d_f0fd6a80bdbc.slice/crio-1d5cd9c0a3ca8f3af6b0e02fb80fac40ad751a32abe1697b01665310ba58fc4b WatchSource:0}: Error finding container 1d5cd9c0a3ca8f3af6b0e02fb80fac40ad751a32abe1697b01665310ba58fc4b: Status 404 returned error can't find the container with id 1d5cd9c0a3ca8f3af6b0e02fb80fac40ad751a32abe1697b01665310ba58fc4b Nov 02 20:29:44.615016 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.614952 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" podStartSLOduration=1.614935267 podStartE2EDuration="1.614935267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:44.613901357 +0000 UTC m=+2227.224553449" watchObservedRunningTime="2026-11-02 20:29:44.614935267 +0000 UTC m=+2227.225587359" Nov 02 20:29:44.832297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.832240 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k"] Nov 02 20:29:44.832647 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.832617 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" podUID="87528081-027a-4246-8cf4-70dbea59d933" containerName="istio-proxy" containerID="cri-o://e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36" gracePeriod=30 Nov 02 20:29:44.839518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.839491 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57"] Nov 02 20:29:44.839762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:44.839736 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" podUID="71f8a8a3-fd48-43d2-bf61-87ed31128081" containerName="istio-proxy" containerID="cri-o://4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85" gracePeriod=30 Nov 02 20:29:45.516294 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.516243 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" event={"ID":"67e804bb-46a1-4963-af5d-f0fd6a80bdbc","Type":"ContainerStarted","Data":"abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d"} Nov 02 20:29:45.516512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.516334 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:29:45.516512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.516352 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" event={"ID":"67e804bb-46a1-4963-af5d-f0fd6a80bdbc","Type":"ContainerStarted","Data":"1d5cd9c0a3ca8f3af6b0e02fb80fac40ad751a32abe1697b01665310ba58fc4b"} Nov 02 20:29:45.624525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.624472 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" podStartSLOduration=2.624455606 podStartE2EDuration="2.624455606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:45.621968756 +0000 UTC m=+2228.232620850" watchObservedRunningTime="2026-11-02 20:29:45.624455606 +0000 UTC m=+2228.235107691" Nov 02 20:29:45.659801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.659767 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv"] Nov 02 20:29:45.667035 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.667003 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:45.716756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.716723 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:29:45.716756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.716723 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:29:45.718690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.718660 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv"] Nov 02 20:29:45.823537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.821783 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:45.823537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.821863 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:45.823537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.822041 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqxf\" (UniqueName: \"kubernetes.io/projected/8acfc830-3a5f-43ff-a28f-de8032f61754-kube-api-access-ptqxf\") pod \"maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:45.924086 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.924048 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:45.924338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.924109 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:45.924338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.924131 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqxf\" (UniqueName: \"kubernetes.io/projected/8acfc830-3a5f-43ff-a28f-de8032f61754-kube-api-access-ptqxf\") pod \"maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:45.924831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.924806 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:45.926551 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.926528 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:45.983887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.983010 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqxf\" (UniqueName: \"kubernetes.io/projected/8acfc830-3a5f-43ff-a28f-de8032f61754-kube-api-access-ptqxf\") pod \"maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:45.984072 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:45.983956 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:46.360124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:46.360094 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv"] Nov 02 20:29:46.360388 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:29:46.360354 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acfc830_3a5f_43ff_a28f_de8032f61754.slice/crio-addf90ba9fcb1c06bb6ebc213e809fc8c31a451cd1b94c368b72bb1e3c9c80aa WatchSource:0}: Error finding container addf90ba9fcb1c06bb6ebc213e809fc8c31a451cd1b94c368b72bb1e3c9c80aa: Status 404 returned error can't find the container with id addf90ba9fcb1c06bb6ebc213e809fc8c31a451cd1b94c368b72bb1e3c9c80aa Nov 02 20:29:46.525147 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:46.525112 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" event={"ID":"8acfc830-3a5f-43ff-a28f-de8032f61754","Type":"ContainerStarted","Data":"addf90ba9fcb1c06bb6ebc213e809fc8c31a451cd1b94c368b72bb1e3c9c80aa"} Nov 02 20:29:47.534288 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:47.534233 5399 generic.go:356] "Generic (PLEG): container finished" podID="8acfc830-3a5f-43ff-a28f-de8032f61754" containerID="ae397de6b49e3622e87cd48497aface58c97b02f8188e9bcdd120adbd876fb8c" exitCode=0 Nov 02 20:29:47.534659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:47.534316 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" event={"ID":"8acfc830-3a5f-43ff-a28f-de8032f61754","Type":"ContainerDied","Data":"ae397de6b49e3622e87cd48497aface58c97b02f8188e9bcdd120adbd876fb8c"} Nov 02 20:29:48.239015 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:29:48.238975 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:29:48.705494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:48.705465 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:48.867123 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:48.867084 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-openshift-ca\") pod \"8acfc830-3a5f-43ff-a28f-de8032f61754\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " Nov 02 20:29:48.867296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:48.867180 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-xks-ca\") pod \"8acfc830-3a5f-43ff-a28f-de8032f61754\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " Nov 02 20:29:48.867382 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:48.867367 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8acfc830-3a5f-43ff-a28f-de8032f61754-kube-api-access-ptqxf\" (UniqueName: \"kubernetes.io/projected/8acfc830-3a5f-43ff-a28f-de8032f61754-kube-api-access-ptqxf\") pod \"8acfc830-3a5f-43ff-a28f-de8032f61754\" (UID: \"8acfc830-3a5f-43ff-a28f-de8032f61754\") " Nov 02 20:29:48.867458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:48.867431 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-openshift-ca" pod "8acfc830-3a5f-43ff-a28f-de8032f61754" (UID: "8acfc830-3a5f-43ff-a28f-de8032f61754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:48.868084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:48.868056 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:48.869356 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:48.869321 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-xks-ca" pod "8acfc830-3a5f-43ff-a28f-de8032f61754" (UID: "8acfc830-3a5f-43ff-a28f-de8032f61754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:48.869470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:48.869438 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acfc830-3a5f-43ff-a28f-de8032f61754-kube-api-access-ptqxf" pod "8acfc830-3a5f-43ff-a28f-de8032f61754" (UID: "8acfc830-3a5f-43ff-a28f-de8032f61754"). InnerVolumeSpecName "kube-api-access-ptqxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:48.970057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:48.970015 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acfc830-3a5f-43ff-a28f-de8032f61754-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:48.970057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:48.970058 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqxf\" (UniqueName: \"kubernetes.io/projected/8acfc830-3a5f-43ff-a28f-de8032f61754-kube-api-access-ptqxf\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:49.552942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:49.552908 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" Nov 02 20:29:49.553104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:49.552936 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv" event={"ID":"8acfc830-3a5f-43ff-a28f-de8032f61754","Type":"ContainerDied","Data":"addf90ba9fcb1c06bb6ebc213e809fc8c31a451cd1b94c368b72bb1e3c9c80aa"} Nov 02 20:29:49.553104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:49.552991 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="addf90ba9fcb1c06bb6ebc213e809fc8c31a451cd1b94c368b72bb1e3c9c80aa" Nov 02 20:29:50.280092 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.280063 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:29:50.284410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.284384 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-envoy\") pod \"87528081-027a-4246-8cf4-70dbea59d933\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " Nov 02 20:29:50.284549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.284428 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87528081-027a-4246-8cf4-70dbea59d933-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87528081-027a-4246-8cf4-70dbea59d933-istio-podinfo\") pod \"87528081-027a-4246-8cf4-70dbea59d933\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " Nov 02 20:29:50.284549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.284465 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-kube-api-access-xqm68\" (UniqueName: \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-kube-api-access-xqm68\") pod \"87528081-027a-4246-8cf4-70dbea59d933\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " Nov 02 20:29:50.284549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.284530 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-socket\") pod \"87528081-027a-4246-8cf4-70dbea59d933\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " Nov 02 20:29:50.284720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.284554 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-data\") pod \"87528081-027a-4246-8cf4-70dbea59d933\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " Nov 02 20:29:50.284720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.284573 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-credential-socket\") pod \"87528081-027a-4246-8cf4-70dbea59d933\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " Nov 02 20:29:50.284720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.284714 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-certs\") pod \"87528081-027a-4246-8cf4-70dbea59d933\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " Nov 02 20:29:50.284873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.284749 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-istio-token\" (UniqueName: \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-istio-token\") pod \"87528081-027a-4246-8cf4-70dbea59d933\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " Nov 02 20:29:50.284873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.284795 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87528081-027a-4246-8cf4-70dbea59d933-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87528081-027a-4246-8cf4-70dbea59d933-istiod-ca-cert\") pod \"87528081-027a-4246-8cf4-70dbea59d933\" (UID: \"87528081-027a-4246-8cf4-70dbea59d933\") " Nov 02 20:29:50.285477 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.285371 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-certs" pod "87528081-027a-4246-8cf4-70dbea59d933" (UID: "87528081-027a-4246-8cf4-70dbea59d933"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.285594 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.285521 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-socket" pod "87528081-027a-4246-8cf4-70dbea59d933" (UID: "87528081-027a-4246-8cf4-70dbea59d933"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.285854 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.285664 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87528081-027a-4246-8cf4-70dbea59d933-istiod-ca-cert" pod "87528081-027a-4246-8cf4-70dbea59d933" (UID: "87528081-027a-4246-8cf4-70dbea59d933"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:50.285854 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.285736 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-data" pod "87528081-027a-4246-8cf4-70dbea59d933" (UID: "87528081-027a-4246-8cf4-70dbea59d933"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.285854 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.285811 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.285854 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.285833 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.286193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.286163 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-credential-socket" pod "87528081-027a-4246-8cf4-70dbea59d933" (UID: "87528081-027a-4246-8cf4-70dbea59d933"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.288028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.287999 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87528081-027a-4246-8cf4-70dbea59d933-istio-podinfo" pod "87528081-027a-4246-8cf4-70dbea59d933" (UID: "87528081-027a-4246-8cf4-70dbea59d933"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:29:50.301478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.301446 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-envoy" pod "87528081-027a-4246-8cf4-70dbea59d933" (UID: "87528081-027a-4246-8cf4-70dbea59d933"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.301607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.301492 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-kube-api-access-xqm68" pod "87528081-027a-4246-8cf4-70dbea59d933" (UID: "87528081-027a-4246-8cf4-70dbea59d933"). InnerVolumeSpecName "kube-api-access-xqm68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:50.303160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.303141 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-istio-token" pod "87528081-027a-4246-8cf4-70dbea59d933" (UID: "87528081-027a-4246-8cf4-70dbea59d933"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:50.330907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.330883 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:29:50.386495 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386465 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-certs\") pod \"71f8a8a3-fd48-43d2-bf61-87ed31128081\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " Nov 02 20:29:50.386635 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386504 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-podinfo\") pod \"71f8a8a3-fd48-43d2-bf61-87ed31128081\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " Nov 02 20:29:50.386635 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386533 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-data\") pod \"71f8a8a3-fd48-43d2-bf61-87ed31128081\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " Nov 02 20:29:50.386635 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386574 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-kube-api-access-sc5z9\" (UniqueName: \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-kube-api-access-sc5z9\") pod \"71f8a8a3-fd48-43d2-bf61-87ed31128081\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " Nov 02 20:29:50.386798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386660 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-token\" (UniqueName: \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-token\") pod \"71f8a8a3-fd48-43d2-bf61-87ed31128081\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " Nov 02 20:29:50.386798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386679 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f8a8a3-fd48-43d2-bf61-87ed31128081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f8a8a3-fd48-43d2-bf61-87ed31128081-istiod-ca-cert\") pod \"71f8a8a3-fd48-43d2-bf61-87ed31128081\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " Nov 02 20:29:50.386798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386735 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-credential-socket\") pod \"71f8a8a3-fd48-43d2-bf61-87ed31128081\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " Nov 02 20:29:50.386798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386735 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-certs" pod "71f8a8a3-fd48-43d2-bf61-87ed31128081" (UID: "71f8a8a3-fd48-43d2-bf61-87ed31128081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.386798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386771 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-socket\") pod \"71f8a8a3-fd48-43d2-bf61-87ed31128081\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " Nov 02 20:29:50.387041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386845 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-envoy\") pod \"71f8a8a3-fd48-43d2-bf61-87ed31128081\" (UID: \"71f8a8a3-fd48-43d2-bf61-87ed31128081\") " Nov 02 20:29:50.387041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.386863 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-data" pod "71f8a8a3-fd48-43d2-bf61-87ed31128081" (UID: "71f8a8a3-fd48-43d2-bf61-87ed31128081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.387168 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387030 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-credential-socket" pod "71f8a8a3-fd48-43d2-bf61-87ed31128081" (UID: "71f8a8a3-fd48-43d2-bf61-87ed31128081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.387168 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387084 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-socket" pod "71f8a8a3-fd48-43d2-bf61-87ed31128081" (UID: "71f8a8a3-fd48-43d2-bf61-87ed31128081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.387168 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387110 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f8a8a3-fd48-43d2-bf61-87ed31128081-istiod-ca-cert" pod "71f8a8a3-fd48-43d2-bf61-87ed31128081" (UID: "71f8a8a3-fd48-43d2-bf61-87ed31128081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:50.387573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387554 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387573 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387582 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387591 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387600 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387611 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87528081-027a-4246-8cf4-70dbea59d933-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387624 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71f8a8a3-fd48-43d2-bf61-87ed31128081-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387634 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387644 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387654 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87528081-027a-4246-8cf4-70dbea59d933-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387663 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87528081-027a-4246-8cf4-70dbea59d933-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.387747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.387698 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqm68\" (UniqueName: \"kubernetes.io/projected/87528081-027a-4246-8cf4-70dbea59d933-kube-api-access-xqm68\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.388916 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.388894 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-token" pod "71f8a8a3-fd48-43d2-bf61-87ed31128081" (UID: "71f8a8a3-fd48-43d2-bf61-87ed31128081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:50.388998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.388970 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-kube-api-access-sc5z9" pod "71f8a8a3-fd48-43d2-bf61-87ed31128081" (UID: "71f8a8a3-fd48-43d2-bf61-87ed31128081"). InnerVolumeSpecName "kube-api-access-sc5z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:50.389178 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.389164 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-podinfo" pod "71f8a8a3-fd48-43d2-bf61-87ed31128081" (UID: "71f8a8a3-fd48-43d2-bf61-87ed31128081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:29:50.389214 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.389177 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-envoy" pod "71f8a8a3-fd48-43d2-bf61-87ed31128081" (UID: "71f8a8a3-fd48-43d2-bf61-87ed31128081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:29:50.488943 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.488907 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.488943 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.488941 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.489159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.488956 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71f8a8a3-fd48-43d2-bf61-87ed31128081-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.489159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.488971 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc5z9\" (UniqueName: \"kubernetes.io/projected/71f8a8a3-fd48-43d2-bf61-87ed31128081-kube-api-access-sc5z9\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:50.564083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.564047 5399 generic.go:356] "Generic (PLEG): container finished" podID="71f8a8a3-fd48-43d2-bf61-87ed31128081" containerID="4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85" exitCode=0 Nov 02 20:29:50.564292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.564100 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" Nov 02 20:29:50.564292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.564136 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" event={"ID":"71f8a8a3-fd48-43d2-bf61-87ed31128081","Type":"ContainerDied","Data":"4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85"} Nov 02 20:29:50.564292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.564206 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57" event={"ID":"71f8a8a3-fd48-43d2-bf61-87ed31128081","Type":"ContainerDied","Data":"7a9b9da9bb62087e7b68b5e6918b48b08364e7189919b2f3571118c14dd35e16"} Nov 02 20:29:50.564292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.564240 5399 scope.go:122] "RemoveContainer" containerID="4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85" Nov 02 20:29:50.565782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.565760 5399 generic.go:356] "Generic (PLEG): container finished" podID="87528081-027a-4246-8cf4-70dbea59d933" containerID="e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36" exitCode=0 Nov 02 20:29:50.565906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.565812 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" Nov 02 20:29:50.565906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.565848 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" event={"ID":"87528081-027a-4246-8cf4-70dbea59d933","Type":"ContainerDied","Data":"e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36"} Nov 02 20:29:50.565906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.565903 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k" event={"ID":"87528081-027a-4246-8cf4-70dbea59d933","Type":"ContainerDied","Data":"d687204fadc7abf299ecf023d135435fbd67feaaa41cf346400beb5f8ce9fcfb"} Nov 02 20:29:50.579325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.579310 5399 scope.go:122] "RemoveContainer" containerID="4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85" Nov 02 20:29:50.579615 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:29:50.579597 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85\": container with ID starting with 4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85 not found: ID does not exist" containerID="4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85" Nov 02 20:29:50.579684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.579625 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85"} err="failed to get container status \"4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85\": rpc error: code = NotFound desc = could not find container \"4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85\": container with ID starting with 4dbdc57b3ae8270a469c9214a8d53bcf583998efb287a61017c24498643ffe85 not found: ID does not exist" Nov 02 20:29:50.579684 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.579651 5399 scope.go:122] "RemoveContainer" containerID="e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36" Nov 02 20:29:50.592184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.592162 5399 scope.go:122] "RemoveContainer" containerID="e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36" Nov 02 20:29:50.592574 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:29:50.592556 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36\": container with ID starting with e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36 not found: ID does not exist" containerID="e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36" Nov 02 20:29:50.592644 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.592581 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36"} err="failed to get container status \"e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36\": rpc error: code = NotFound desc = could not find container \"e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36\": container with ID starting with e6fe07da35a81fd44f33125618703a08e67ab3cdc1f90d983ff5a6b6b342ab36 not found: ID does not exist" Nov 02 20:29:50.735936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.735897 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k"] Nov 02 20:29:50.743244 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.743218 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-fvw5k"] Nov 02 20:29:50.858732 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.858656 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57"] Nov 02 20:29:50.865693 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:50.865668 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7dcb03-openshift-default-5687b46945-7xd57"] Nov 02 20:29:51.512869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:51.512832 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f8a8a3-fd48-43d2-bf61-87ed31128081" path="/var/lib/kubelet/pods/71f8a8a3-fd48-43d2-bf61-87ed31128081/volumes" Nov 02 20:29:51.513443 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:51.513422 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87528081-027a-4246-8cf4-70dbea59d933" path="/var/lib/kubelet/pods/87528081-027a-4246-8cf4-70dbea59d933/volumes" Nov 02 20:29:55.519263 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:55.519234 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:29:56.528560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:56.528530 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:29:57.290757 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:29:57.290713 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl"] Nov 02 20:30:00.164148 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.164112 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj"] Nov 02 20:30:00.165029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.165007 5399 cpu_manager.go:414] "containerMap: removing container" podUID="8acfc830-3a5f-43ff-a28f-de8032f61754" containerName="revoke-keys" Nov 02 20:30:00.165088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.165033 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8acfc830-3a5f-43ff-a28f-de8032f61754" containerName="revoke-keys" Nov 02 20:30:00.165088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.165048 5399 cpu_manager.go:414] "containerMap: removing container" podUID="87528081-027a-4246-8cf4-70dbea59d933" containerName="istio-proxy" Nov 02 20:30:00.165088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.165055 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87528081-027a-4246-8cf4-70dbea59d933" containerName="istio-proxy" Nov 02 20:30:00.165198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.165095 5399 cpu_manager.go:414] "containerMap: removing container" podUID="71f8a8a3-fd48-43d2-bf61-87ed31128081" containerName="istio-proxy" Nov 02 20:30:00.165198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.165103 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f8a8a3-fd48-43d2-bf61-87ed31128081" containerName="istio-proxy" Nov 02 20:30:00.165298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.165260 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="8acfc830-3a5f-43ff-a28f-de8032f61754" containerName="revoke-keys" Nov 02 20:30:00.165298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.165293 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="87528081-027a-4246-8cf4-70dbea59d933" containerName="istio-proxy" Nov 02 20:30:00.165369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.165310 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f8a8a3-fd48-43d2-bf61-87ed31128081" containerName="istio-proxy" Nov 02 20:30:00.170907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.170885 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n"] Nov 02 20:30:00.171031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.170995 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" Nov 02 20:30:00.175238 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.175215 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8"] Nov 02 20:30:00.175376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.175342 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" Nov 02 20:30:00.179264 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.179244 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" Nov 02 20:30:00.210102 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.210070 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh"] Nov 02 20:30:00.214877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.214854 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9"] Nov 02 20:30:00.215058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.214970 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" Nov 02 20:30:00.220031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.220008 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws"] Nov 02 20:30:00.220310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.220224 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" Nov 02 20:30:00.224430 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.224411 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr"] Nov 02 20:30:00.224562 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.224502 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" Nov 02 20:30:00.232089 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.232048 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8"] Nov 02 20:30:00.233006 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.232984 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" Nov 02 20:30:00.249963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.249502 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" Nov 02 20:30:00.251826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.251800 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:30:00.258519 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.258498 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt"] Nov 02 20:30:00.262980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.262956 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" Nov 02 20:30:00.263649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.263624 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc569dc-8a4b-4774-9b77-a020e1d821a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj\" (UID: \"3fc569dc-8a4b-4774-9b77-a020e1d821a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" Nov 02 20:30:00.263795 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.263779 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6dh\" (UniqueName: \"kubernetes.io/projected/3fc569dc-8a4b-4774-9b77-a020e1d821a2-kube-api-access-mg6dh\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj\" (UID: \"3fc569dc-8a4b-4774-9b77-a020e1d821a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" Nov 02 20:30:00.264578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.264555 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8"] Nov 02 20:30:00.298501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.297624 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh"] Nov 02 20:30:00.306292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.303737 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n"] Nov 02 20:30:00.312025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.310025 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9"] Nov 02 20:30:00.312747 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.312718 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr"] Nov 02 20:30:00.347768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.347732 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8"] Nov 02 20:30:00.359311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.358429 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj"] Nov 02 20:30:00.365662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365634 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6mb\" (UniqueName: \"kubernetes.io/projected/dd0d8e94-5394-46a9-aceb-08532e885e00-kube-api-access-qm6mb\") pod \"maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr\" (UID: \"dd0d8e94-5394-46a9-aceb-08532e885e00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" Nov 02 20:30:00.365821 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365680 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc569dc-8a4b-4774-9b77-a020e1d821a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj\" (UID: \"3fc569dc-8a4b-4774-9b77-a020e1d821a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" Nov 02 20:30:00.365821 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365704 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzsh\" (UniqueName: \"kubernetes.io/projected/d104368e-b2cc-4929-81b0-48a99a44c29f-kube-api-access-qjzsh\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9\" (UID: \"d104368e-b2cc-4929-81b0-48a99a44c29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" Nov 02 20:30:00.365821 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365723 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3621789-faf8-482e-bf9d-76caea94a3e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh\" (UID: \"c3621789-faf8-482e-bf9d-76caea94a3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" Nov 02 20:30:00.365821 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365747 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d8qb\" (UniqueName: \"kubernetes.io/projected/f8035ece-b691-4025-aab3-182741493aa0-kube-api-access-9d8qb\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt\" (UID: \"f8035ece-b691-4025-aab3-182741493aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" Nov 02 20:30:00.365821 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365765 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e81e4f5-0154-4327-9de3-6690783b632a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws\" (UID: \"9e81e4f5-0154-4327-9de3-6690783b632a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" Nov 02 20:30:00.365821 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365794 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcw9s\" (UniqueName: \"kubernetes.io/projected/29bbab00-c103-433f-a3ec-90d6b1faec7b-kube-api-access-gcw9s\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8\" (UID: \"29bbab00-c103-433f-a3ec-90d6b1faec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" Nov 02 20:30:00.366161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365871 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bbab00-c103-433f-a3ec-90d6b1faec7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8\" (UID: \"29bbab00-c103-433f-a3ec-90d6b1faec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" Nov 02 20:30:00.366161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365891 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d104368e-b2cc-4929-81b0-48a99a44c29f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9\" (UID: \"d104368e-b2cc-4929-81b0-48a99a44c29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" Nov 02 20:30:00.366161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365925 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6dh\" (UniqueName: \"kubernetes.io/projected/3fc569dc-8a4b-4774-9b77-a020e1d821a2-kube-api-access-mg6dh\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj\" (UID: \"3fc569dc-8a4b-4774-9b77-a020e1d821a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" Nov 02 20:30:00.366161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365977 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhjw\" (UniqueName: \"kubernetes.io/projected/5eda0988-9984-43df-afa4-5ce94fa58c5a-kube-api-access-plhjw\") pod \"maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8\" (UID: \"5eda0988-9984-43df-afa4-5ce94fa58c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" Nov 02 20:30:00.366161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.365997 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfv8\" (UniqueName: \"kubernetes.io/projected/e254e9b3-e008-4c18-b6b1-786188414fa9-kube-api-access-7mfv8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n\" (UID: \"e254e9b3-e008-4c18-b6b1-786188414fa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" Nov 02 20:30:00.366161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.366029 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8035ece-b691-4025-aab3-182741493aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt\" (UID: \"f8035ece-b691-4025-aab3-182741493aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" Nov 02 20:30:00.366161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.366061 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghrkh\" (UniqueName: \"kubernetes.io/projected/9e81e4f5-0154-4327-9de3-6690783b632a-kube-api-access-ghrkh\") pod \"maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws\" (UID: \"9e81e4f5-0154-4327-9de3-6690783b632a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" Nov 02 20:30:00.366161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.366084 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s688\" (UniqueName: \"kubernetes.io/projected/c3621789-faf8-482e-bf9d-76caea94a3e2-kube-api-access-8s688\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh\" (UID: \"c3621789-faf8-482e-bf9d-76caea94a3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" Nov 02 20:30:00.366161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.366103 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0d8e94-5394-46a9-aceb-08532e885e00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr\" (UID: \"dd0d8e94-5394-46a9-aceb-08532e885e00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" Nov 02 20:30:00.366803 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.366781 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc569dc-8a4b-4774-9b77-a020e1d821a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj\" (UID: \"3fc569dc-8a4b-4774-9b77-a020e1d821a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" Nov 02 20:30:00.367109 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.367087 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e254e9b3-e008-4c18-b6b1-786188414fa9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n\" (UID: \"e254e9b3-e008-4c18-b6b1-786188414fa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" Nov 02 20:30:00.367194 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.367149 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda0988-9984-43df-afa4-5ce94fa58c5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8\" (UID: \"5eda0988-9984-43df-afa4-5ce94fa58c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" Nov 02 20:30:00.374626 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.374601 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws"] Nov 02 20:30:00.418948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.418866 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt"] Nov 02 20:30:00.425464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.425434 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6dh\" (UniqueName: \"kubernetes.io/projected/3fc569dc-8a4b-4774-9b77-a020e1d821a2-kube-api-access-mg6dh\") pod \"maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj\" (UID: \"3fc569dc-8a4b-4774-9b77-a020e1d821a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" Nov 02 20:30:00.468410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.468377 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d8qb\" (UniqueName: \"kubernetes.io/projected/f8035ece-b691-4025-aab3-182741493aa0-kube-api-access-9d8qb\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt\" (UID: \"f8035ece-b691-4025-aab3-182741493aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" Nov 02 20:30:00.468410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.468415 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e81e4f5-0154-4327-9de3-6690783b632a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws\" (UID: \"9e81e4f5-0154-4327-9de3-6690783b632a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" Nov 02 20:30:00.468659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.468445 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcw9s\" (UniqueName: \"kubernetes.io/projected/29bbab00-c103-433f-a3ec-90d6b1faec7b-kube-api-access-gcw9s\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8\" (UID: \"29bbab00-c103-433f-a3ec-90d6b1faec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" Nov 02 20:30:00.468659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.468492 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bbab00-c103-433f-a3ec-90d6b1faec7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8\" (UID: \"29bbab00-c103-433f-a3ec-90d6b1faec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" Nov 02 20:30:00.468659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.468507 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d104368e-b2cc-4929-81b0-48a99a44c29f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9\" (UID: \"d104368e-b2cc-4929-81b0-48a99a44c29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" Nov 02 20:30:00.468659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.468573 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhjw\" (UniqueName: \"kubernetes.io/projected/5eda0988-9984-43df-afa4-5ce94fa58c5a-kube-api-access-plhjw\") pod \"maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8\" (UID: \"5eda0988-9984-43df-afa4-5ce94fa58c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" Nov 02 20:30:00.468659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.468601 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfv8\" (UniqueName: \"kubernetes.io/projected/e254e9b3-e008-4c18-b6b1-786188414fa9-kube-api-access-7mfv8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n\" (UID: \"e254e9b3-e008-4c18-b6b1-786188414fa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" Nov 02 20:30:00.468930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.468902 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8035ece-b691-4025-aab3-182741493aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt\" (UID: \"f8035ece-b691-4025-aab3-182741493aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" Nov 02 20:30:00.468987 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.468970 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghrkh\" (UniqueName: \"kubernetes.io/projected/9e81e4f5-0154-4327-9de3-6690783b632a-kube-api-access-ghrkh\") pod \"maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws\" (UID: \"9e81e4f5-0154-4327-9de3-6690783b632a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" Nov 02 20:30:00.469043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469014 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s688\" (UniqueName: \"kubernetes.io/projected/c3621789-faf8-482e-bf9d-76caea94a3e2-kube-api-access-8s688\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh\" (UID: \"c3621789-faf8-482e-bf9d-76caea94a3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" Nov 02 20:30:00.469105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469061 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0d8e94-5394-46a9-aceb-08532e885e00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr\" (UID: \"dd0d8e94-5394-46a9-aceb-08532e885e00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" Nov 02 20:30:00.469105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469090 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e81e4f5-0154-4327-9de3-6690783b632a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws\" (UID: \"9e81e4f5-0154-4327-9de3-6690783b632a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" Nov 02 20:30:00.469344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469314 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bbab00-c103-433f-a3ec-90d6b1faec7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8\" (UID: \"29bbab00-c103-433f-a3ec-90d6b1faec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" Nov 02 20:30:00.469344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469340 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d104368e-b2cc-4929-81b0-48a99a44c29f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9\" (UID: \"d104368e-b2cc-4929-81b0-48a99a44c29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" Nov 02 20:30:00.469533 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469516 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e254e9b3-e008-4c18-b6b1-786188414fa9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n\" (UID: \"e254e9b3-e008-4c18-b6b1-786188414fa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" Nov 02 20:30:00.469570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469560 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda0988-9984-43df-afa4-5ce94fa58c5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8\" (UID: \"5eda0988-9984-43df-afa4-5ce94fa58c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" Nov 02 20:30:00.469657 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469633 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6mb\" (UniqueName: \"kubernetes.io/projected/dd0d8e94-5394-46a9-aceb-08532e885e00-kube-api-access-qm6mb\") pod \"maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr\" (UID: \"dd0d8e94-5394-46a9-aceb-08532e885e00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" Nov 02 20:30:00.469758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469702 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzsh\" (UniqueName: \"kubernetes.io/projected/d104368e-b2cc-4929-81b0-48a99a44c29f-kube-api-access-qjzsh\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9\" (UID: \"d104368e-b2cc-4929-81b0-48a99a44c29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" Nov 02 20:30:00.469758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469725 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8035ece-b691-4025-aab3-182741493aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt\" (UID: \"f8035ece-b691-4025-aab3-182741493aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" Nov 02 20:30:00.469758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.469733 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3621789-faf8-482e-bf9d-76caea94a3e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh\" (UID: \"c3621789-faf8-482e-bf9d-76caea94a3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" Nov 02 20:30:00.470034 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.470013 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0d8e94-5394-46a9-aceb-08532e885e00-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr\" (UID: \"dd0d8e94-5394-46a9-aceb-08532e885e00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" Nov 02 20:30:00.470230 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.470211 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda0988-9984-43df-afa4-5ce94fa58c5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8\" (UID: \"5eda0988-9984-43df-afa4-5ce94fa58c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" Nov 02 20:30:00.470291 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.470218 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e254e9b3-e008-4c18-b6b1-786188414fa9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n\" (UID: \"e254e9b3-e008-4c18-b6b1-786188414fa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" Nov 02 20:30:00.470397 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.470381 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3621789-faf8-482e-bf9d-76caea94a3e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh\" (UID: \"c3621789-faf8-482e-bf9d-76caea94a3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" Nov 02 20:30:00.487314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.487292 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" Nov 02 20:30:00.522751 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.522714 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghrkh\" (UniqueName: \"kubernetes.io/projected/9e81e4f5-0154-4327-9de3-6690783b632a-kube-api-access-ghrkh\") pod \"maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws\" (UID: \"9e81e4f5-0154-4327-9de3-6690783b632a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" Nov 02 20:30:00.523423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.523396 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcw9s\" (UniqueName: \"kubernetes.io/projected/29bbab00-c103-433f-a3ec-90d6b1faec7b-kube-api-access-gcw9s\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8\" (UID: \"29bbab00-c103-433f-a3ec-90d6b1faec7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" Nov 02 20:30:00.524149 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.524074 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s688\" (UniqueName: \"kubernetes.io/projected/c3621789-faf8-482e-bf9d-76caea94a3e2-kube-api-access-8s688\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh\" (UID: \"c3621789-faf8-482e-bf9d-76caea94a3e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" Nov 02 20:30:00.524265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.524158 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d8qb\" (UniqueName: \"kubernetes.io/projected/f8035ece-b691-4025-aab3-182741493aa0-kube-api-access-9d8qb\") pod \"maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt\" (UID: \"f8035ece-b691-4025-aab3-182741493aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" Nov 02 20:30:00.524367 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.524345 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfv8\" (UniqueName: \"kubernetes.io/projected/e254e9b3-e008-4c18-b6b1-786188414fa9-kube-api-access-7mfv8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n\" (UID: \"e254e9b3-e008-4c18-b6b1-786188414fa9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" Nov 02 20:30:00.524693 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.524664 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6mb\" (UniqueName: \"kubernetes.io/projected/dd0d8e94-5394-46a9-aceb-08532e885e00-kube-api-access-qm6mb\") pod \"maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr\" (UID: \"dd0d8e94-5394-46a9-aceb-08532e885e00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" Nov 02 20:30:00.525521 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.525455 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhjw\" (UniqueName: \"kubernetes.io/projected/5eda0988-9984-43df-afa4-5ce94fa58c5a-kube-api-access-plhjw\") pod \"maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8\" (UID: \"5eda0988-9984-43df-afa4-5ce94fa58c5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" Nov 02 20:30:00.527227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.527191 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzsh\" (UniqueName: \"kubernetes.io/projected/d104368e-b2cc-4929-81b0-48a99a44c29f-kube-api-access-qjzsh\") pod \"maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9\" (UID: \"d104368e-b2cc-4929-81b0-48a99a44c29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" Nov 02 20:30:00.536173 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.536007 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" Nov 02 20:30:00.549760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.549728 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" Nov 02 20:30:00.561959 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.561772 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" Nov 02 20:30:00.571977 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.571805 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" Nov 02 20:30:00.579726 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.579550 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" Nov 02 20:30:00.592707 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.592300 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" Nov 02 20:30:00.797730 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.797688 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" Nov 02 20:30:00.807405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.807376 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" Nov 02 20:30:00.871338 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:30:00.871306 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc569dc_8a4b_4774_9b77_a020e1d821a2.slice/crio-644b95c42750d56920453eb237a8c55cef473464c70f2f47d855e39755249f66 WatchSource:0}: Error finding container 644b95c42750d56920453eb237a8c55cef473464c70f2f47d855e39755249f66: Status 404 returned error can't find the container with id 644b95c42750d56920453eb237a8c55cef473464c70f2f47d855e39755249f66 Nov 02 20:30:00.875605 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.875077 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj"] Nov 02 20:30:00.945838 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:30:00.945673 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3621789_faf8_482e_bf9d_76caea94a3e2.slice/crio-43db0d78e5d7ee65a8d1370fcaa9ee8350b8a15424a2f64ef3a2574b6cbbb521 WatchSource:0}: Error finding container 43db0d78e5d7ee65a8d1370fcaa9ee8350b8a15424a2f64ef3a2574b6cbbb521: Status 404 returned error can't find the container with id 43db0d78e5d7ee65a8d1370fcaa9ee8350b8a15424a2f64ef3a2574b6cbbb521 Nov 02 20:30:00.949250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.949222 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh"] Nov 02 20:30:00.964939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:00.964875 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9"] Nov 02 20:30:00.967053 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:30:00.967028 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd104368e_b2cc_4929_81b0_48a99a44c29f.slice/crio-73da3130e07c27417163fbb468839ff57cc50ab13e947e1b8545217208b7d9c0 WatchSource:0}: Error finding container 73da3130e07c27417163fbb468839ff57cc50ab13e947e1b8545217208b7d9c0: Status 404 returned error can't find the container with id 73da3130e07c27417163fbb468839ff57cc50ab13e947e1b8545217208b7d9c0 Nov 02 20:30:01.001552 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.001520 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws"] Nov 02 20:30:01.004300 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:30:01.004110 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e81e4f5_0154_4327_9de3_6690783b632a.slice/crio-94ae7753ebfc63a9d5ea2f4778a9becd9b9642753c78e434c257ea04259924dd WatchSource:0}: Error finding container 94ae7753ebfc63a9d5ea2f4778a9becd9b9642753c78e434c257ea04259924dd: Status 404 returned error can't find the container with id 94ae7753ebfc63a9d5ea2f4778a9becd9b9642753c78e434c257ea04259924dd Nov 02 20:30:01.026598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.026571 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8"] Nov 02 20:30:01.029829 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:30:01.029801 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bbab00_c103_433f_a3ec_90d6b1faec7b.slice/crio-a24ee7c184e136076b8fd48eb5bcb2b7c266828dcb4fa9aee8c8971f0fc62c77 WatchSource:0}: Error finding container a24ee7c184e136076b8fd48eb5bcb2b7c266828dcb4fa9aee8c8971f0fc62c77: Status 404 returned error can't find the container with id a24ee7c184e136076b8fd48eb5bcb2b7c266828dcb4fa9aee8c8971f0fc62c77 Nov 02 20:30:01.176420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.176389 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr"] Nov 02 20:30:01.203628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.203603 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n"] Nov 02 20:30:01.213834 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.213806 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8"] Nov 02 20:30:01.231945 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:30:01.231890 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0d8e94_5394_46a9_aceb_08532e885e00.slice/crio-42d4ec103c608b8265919a1c361645a91b59abeb1a3d14835e23a24f43eac341 WatchSource:0}: Error finding container 42d4ec103c608b8265919a1c361645a91b59abeb1a3d14835e23a24f43eac341: Status 404 returned error can't find the container with id 42d4ec103c608b8265919a1c361645a91b59abeb1a3d14835e23a24f43eac341 Nov 02 20:30:01.234382 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:30:01.234269 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode254e9b3_e008_4c18_b6b1_786188414fa9.slice/crio-f6584240fbf2df79cef8202db2f546b29cdc4f4d379f6d737ac4bad156f01d4f WatchSource:0}: Error finding container f6584240fbf2df79cef8202db2f546b29cdc4f4d379f6d737ac4bad156f01d4f: Status 404 returned error can't find the container with id f6584240fbf2df79cef8202db2f546b29cdc4f4d379f6d737ac4bad156f01d4f Nov 02 20:30:01.245385 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.244264 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt"] Nov 02 20:30:01.253805 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:30:01.253761 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8035ece_b691_4025_aab3_182741493aa0.slice/crio-5090dddb7c60cc389c9a5dbfa2e345425791d10b47b684a4e379effe99d6c991 WatchSource:0}: Error finding container 5090dddb7c60cc389c9a5dbfa2e345425791d10b47b684a4e379effe99d6c991: Status 404 returned error can't find the container with id 5090dddb7c60cc389c9a5dbfa2e345425791d10b47b684a4e379effe99d6c991 Nov 02 20:30:01.694778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.694744 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" event={"ID":"f8035ece-b691-4025-aab3-182741493aa0","Type":"ContainerStarted","Data":"5090dddb7c60cc389c9a5dbfa2e345425791d10b47b684a4e379effe99d6c991"} Nov 02 20:30:01.696387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.696305 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" event={"ID":"dd0d8e94-5394-46a9-aceb-08532e885e00","Type":"ContainerStarted","Data":"7f6e7bb4fb03671e84b6061a35692c12084abd371b502351af0e5935c77fee55"} Nov 02 20:30:01.696387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.696363 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" event={"ID":"dd0d8e94-5394-46a9-aceb-08532e885e00","Type":"ContainerStarted","Data":"42d4ec103c608b8265919a1c361645a91b59abeb1a3d14835e23a24f43eac341"} Nov 02 20:30:01.697939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.697914 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" event={"ID":"29bbab00-c103-433f-a3ec-90d6b1faec7b","Type":"ContainerStarted","Data":"c0303b7e123114b9f12f42a0b124f098a37f348a48a1ac7b78ec0ae9e166c85d"} Nov 02 20:30:01.698056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.697961 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" event={"ID":"29bbab00-c103-433f-a3ec-90d6b1faec7b","Type":"ContainerStarted","Data":"a24ee7c184e136076b8fd48eb5bcb2b7c266828dcb4fa9aee8c8971f0fc62c77"} Nov 02 20:30:01.699514 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.699492 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" event={"ID":"d104368e-b2cc-4929-81b0-48a99a44c29f","Type":"ContainerStarted","Data":"f0751ab85fba504d251aad16701dfeed986dad89498a145fa56c64a9b2aa4be2"} Nov 02 20:30:01.699597 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.699528 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" event={"ID":"d104368e-b2cc-4929-81b0-48a99a44c29f","Type":"ContainerStarted","Data":"73da3130e07c27417163fbb468839ff57cc50ab13e947e1b8545217208b7d9c0"} Nov 02 20:30:01.701148 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.701123 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" event={"ID":"5eda0988-9984-43df-afa4-5ce94fa58c5a","Type":"ContainerStarted","Data":"3c9d069d419f7cc3ca5a163b3afa52b219f437ffc675c97501025e9593e666a6"} Nov 02 20:30:01.701254 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.701168 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" event={"ID":"5eda0988-9984-43df-afa4-5ce94fa58c5a","Type":"ContainerStarted","Data":"57afeae69137fb9ce2e4822960f3881ea98bfdfa0f22381cc92bf697d074efdd"} Nov 02 20:30:01.702399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.702378 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" event={"ID":"e254e9b3-e008-4c18-b6b1-786188414fa9","Type":"ContainerStarted","Data":"f6584240fbf2df79cef8202db2f546b29cdc4f4d379f6d737ac4bad156f01d4f"} Nov 02 20:30:01.703969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.703946 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" event={"ID":"c3621789-faf8-482e-bf9d-76caea94a3e2","Type":"ContainerStarted","Data":"043873d70e64a3b14dec3b288adac170f4a64dc49689f0a9260ccf88b0d106e0"} Nov 02 20:30:01.704063 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.703990 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" event={"ID":"c3621789-faf8-482e-bf9d-76caea94a3e2","Type":"ContainerStarted","Data":"43db0d78e5d7ee65a8d1370fcaa9ee8350b8a15424a2f64ef3a2574b6cbbb521"} Nov 02 20:30:01.705597 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.705573 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" event={"ID":"9e81e4f5-0154-4327-9de3-6690783b632a","Type":"ContainerStarted","Data":"bf67a7f78b301632e4c6175cfb4b987b129e1faa3812aefac490ac6adf43dac6"} Nov 02 20:30:01.705699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.705616 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" event={"ID":"9e81e4f5-0154-4327-9de3-6690783b632a","Type":"ContainerStarted","Data":"94ae7753ebfc63a9d5ea2f4778a9becd9b9642753c78e434c257ea04259924dd"} Nov 02 20:30:01.707115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.707095 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" event={"ID":"3fc569dc-8a4b-4774-9b77-a020e1d821a2","Type":"ContainerStarted","Data":"8b0aee22c48c74b879996906d84a058755d0ee22af1dd39575de1af975ba0792"} Nov 02 20:30:01.707205 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.707126 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" event={"ID":"3fc569dc-8a4b-4774-9b77-a020e1d821a2","Type":"ContainerStarted","Data":"644b95c42750d56920453eb237a8c55cef473464c70f2f47d855e39755249f66"} Nov 02 20:30:01.804512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.804448 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" podStartSLOduration=1.804430368 podStartE2EDuration="1.804430368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:01.802614976 +0000 UTC m=+2244.413267156" watchObservedRunningTime="2026-11-02 20:30:01.804430368 +0000 UTC m=+2244.415082461" Nov 02 20:30:01.909929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:01.909867 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" podStartSLOduration=1.909851832 podStartE2EDuration="1.909851832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:01.907422068 +0000 UTC m=+2244.518074172" watchObservedRunningTime="2026-11-02 20:30:01.909851832 +0000 UTC m=+2244.520503962" Nov 02 20:30:02.016243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.016135 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" podStartSLOduration=2.016121941 podStartE2EDuration="2.016121941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.011570125 +0000 UTC m=+2244.622222217" watchObservedRunningTime="2026-11-02 20:30:02.016121941 +0000 UTC m=+2244.626774081" Nov 02 20:30:02.119783 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.119724 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" podStartSLOduration=2.119707219 podStartE2EDuration="2.119707219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.11806655 +0000 UTC m=+2244.728718642" watchObservedRunningTime="2026-11-02 20:30:02.119707219 +0000 UTC m=+2244.730359309" Nov 02 20:30:02.228234 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.228163 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" podStartSLOduration=2.228144807 podStartE2EDuration="2.228144807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.223587134 +0000 UTC m=+2244.834239225" watchObservedRunningTime="2026-11-02 20:30:02.228144807 +0000 UTC m=+2244.838796899" Nov 02 20:30:02.291305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.291239 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" podUID="82d7a94c-d41d-4122-8274-aec68f197f9f" containerName="payload-pre-processing" containerID="cri-o://c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737" gracePeriod=25 Nov 02 20:30:02.334888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.334824 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" podStartSLOduration=2.334803979 podStartE2EDuration="2.334803979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.328658661 +0000 UTC m=+2244.939310782" watchObservedRunningTime="2026-11-02 20:30:02.334803979 +0000 UTC m=+2244.945456071" Nov 02 20:30:02.433964 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.433906 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" podStartSLOduration=2.433889924 podStartE2EDuration="2.433889924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.431941947 +0000 UTC m=+2245.042594040" watchObservedRunningTime="2026-11-02 20:30:02.433889924 +0000 UTC m=+2245.044542015" Nov 02 20:30:02.716206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.716173 5399 generic.go:356] "Generic (PLEG): container finished" podID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerID="8b0aee22c48c74b879996906d84a058755d0ee22af1dd39575de1af975ba0792" exitCode=60 Nov 02 20:30:02.716384 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.716269 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" event={"ID":"3fc569dc-8a4b-4774-9b77-a020e1d821a2","Type":"ContainerDied","Data":"8b0aee22c48c74b879996906d84a058755d0ee22af1dd39575de1af975ba0792"} Nov 02 20:30:02.716608 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.716588 5399 scope.go:122] "RemoveContainer" containerID="8b0aee22c48c74b879996906d84a058755d0ee22af1dd39575de1af975ba0792" Nov 02 20:30:02.717963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.717936 5399 generic.go:356] "Generic (PLEG): container finished" podID="f8035ece-b691-4025-aab3-182741493aa0" containerID="f0ca82211ab20e9ea5d199313581334d546d5e7584855216a48ee836ab128a3f" exitCode=60 Nov 02 20:30:02.718084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.717959 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" event={"ID":"f8035ece-b691-4025-aab3-182741493aa0","Type":"ContainerDied","Data":"f0ca82211ab20e9ea5d199313581334d546d5e7584855216a48ee836ab128a3f"} Nov 02 20:30:02.718255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.718241 5399 scope.go:122] "RemoveContainer" containerID="f0ca82211ab20e9ea5d199313581334d546d5e7584855216a48ee836ab128a3f" Nov 02 20:30:02.720081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.720058 5399 generic.go:356] "Generic (PLEG): container finished" podID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerID="7f6e7bb4fb03671e84b6061a35692c12084abd371b502351af0e5935c77fee55" exitCode=60 Nov 02 20:30:02.720179 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.720164 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" event={"ID":"dd0d8e94-5394-46a9-aceb-08532e885e00","Type":"ContainerDied","Data":"7f6e7bb4fb03671e84b6061a35692c12084abd371b502351af0e5935c77fee55"} Nov 02 20:30:02.720437 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.720420 5399 scope.go:122] "RemoveContainer" containerID="7f6e7bb4fb03671e84b6061a35692c12084abd371b502351af0e5935c77fee55" Nov 02 20:30:02.722075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.722050 5399 generic.go:356] "Generic (PLEG): container finished" podID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerID="c0303b7e123114b9f12f42a0b124f098a37f348a48a1ac7b78ec0ae9e166c85d" exitCode=60 Nov 02 20:30:02.722160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.722124 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" event={"ID":"29bbab00-c103-433f-a3ec-90d6b1faec7b","Type":"ContainerDied","Data":"c0303b7e123114b9f12f42a0b124f098a37f348a48a1ac7b78ec0ae9e166c85d"} Nov 02 20:30:02.722344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.722328 5399 scope.go:122] "RemoveContainer" containerID="c0303b7e123114b9f12f42a0b124f098a37f348a48a1ac7b78ec0ae9e166c85d" Nov 02 20:30:02.723859 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.723836 5399 generic.go:356] "Generic (PLEG): container finished" podID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerID="f0751ab85fba504d251aad16701dfeed986dad89498a145fa56c64a9b2aa4be2" exitCode=60 Nov 02 20:30:02.723948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.723929 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" event={"ID":"d104368e-b2cc-4929-81b0-48a99a44c29f","Type":"ContainerDied","Data":"f0751ab85fba504d251aad16701dfeed986dad89498a145fa56c64a9b2aa4be2"} Nov 02 20:30:02.724203 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.724180 5399 scope.go:122] "RemoveContainer" containerID="f0751ab85fba504d251aad16701dfeed986dad89498a145fa56c64a9b2aa4be2" Nov 02 20:30:02.725637 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.725618 5399 generic.go:356] "Generic (PLEG): container finished" podID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerID="3c9d069d419f7cc3ca5a163b3afa52b219f437ffc675c97501025e9593e666a6" exitCode=60 Nov 02 20:30:02.725740 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.725689 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" event={"ID":"5eda0988-9984-43df-afa4-5ce94fa58c5a","Type":"ContainerDied","Data":"3c9d069d419f7cc3ca5a163b3afa52b219f437ffc675c97501025e9593e666a6"} Nov 02 20:30:02.725868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.725854 5399 scope.go:122] "RemoveContainer" containerID="3c9d069d419f7cc3ca5a163b3afa52b219f437ffc675c97501025e9593e666a6" Nov 02 20:30:02.727208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.727189 5399 generic.go:356] "Generic (PLEG): container finished" podID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerID="3f702b83a245902e04ee4969b7549c23a5b0c335835564729acf3bd263a563f8" exitCode=60 Nov 02 20:30:02.727336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.727208 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" event={"ID":"e254e9b3-e008-4c18-b6b1-786188414fa9","Type":"ContainerDied","Data":"3f702b83a245902e04ee4969b7549c23a5b0c335835564729acf3bd263a563f8"} Nov 02 20:30:02.727527 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.727507 5399 scope.go:122] "RemoveContainer" containerID="3f702b83a245902e04ee4969b7549c23a5b0c335835564729acf3bd263a563f8" Nov 02 20:30:02.727818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.727800 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:30:02.728683 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.728662 5399 generic.go:356] "Generic (PLEG): container finished" podID="82d7a94c-d41d-4122-8274-aec68f197f9f" containerID="c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737" exitCode=0 Nov 02 20:30:02.728827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.728805 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" event={"ID":"82d7a94c-d41d-4122-8274-aec68f197f9f","Type":"ContainerDied","Data":"c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737"} Nov 02 20:30:02.729029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.728952 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" event={"ID":"82d7a94c-d41d-4122-8274-aec68f197f9f","Type":"ContainerDied","Data":"9389c4d4b2f552d75867c4f089ea78accc3d797ac31f74fb43797e43232f8eb3"} Nov 02 20:30:02.729156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.729009 5399 scope.go:122] "RemoveContainer" containerID="c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737" Nov 02 20:30:02.731598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.731577 5399 generic.go:356] "Generic (PLEG): container finished" podID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerID="043873d70e64a3b14dec3b288adac170f4a64dc49689f0a9260ccf88b0d106e0" exitCode=60 Nov 02 20:30:02.731694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.731680 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" event={"ID":"c3621789-faf8-482e-bf9d-76caea94a3e2","Type":"ContainerDied","Data":"043873d70e64a3b14dec3b288adac170f4a64dc49689f0a9260ccf88b0d106e0"} Nov 02 20:30:02.731913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.731894 5399 scope.go:122] "RemoveContainer" containerID="043873d70e64a3b14dec3b288adac170f4a64dc49689f0a9260ccf88b0d106e0" Nov 02 20:30:02.733522 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.733500 5399 generic.go:356] "Generic (PLEG): container finished" podID="9e81e4f5-0154-4327-9de3-6690783b632a" containerID="bf67a7f78b301632e4c6175cfb4b987b129e1faa3812aefac490ac6adf43dac6" exitCode=60 Nov 02 20:30:02.733624 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.733552 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" event={"ID":"9e81e4f5-0154-4327-9de3-6690783b632a","Type":"ContainerDied","Data":"bf67a7f78b301632e4c6175cfb4b987b129e1faa3812aefac490ac6adf43dac6"} Nov 02 20:30:02.733846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.733829 5399 scope.go:122] "RemoveContainer" containerID="bf67a7f78b301632e4c6175cfb4b987b129e1faa3812aefac490ac6adf43dac6" Nov 02 20:30:02.745555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.745533 5399 scope.go:122] "RemoveContainer" containerID="c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737" Nov 02 20:30:02.745839 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:02.745813 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737\": container with ID starting with c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737 not found: ID does not exist" containerID="c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737" Nov 02 20:30:02.745921 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.745847 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737"} err="failed to get container status \"c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737\": rpc error: code = NotFound desc = could not find container \"c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737\": container with ID starting with c0ad651537199cbc4a77dd02910835961b3d4fe772c2adc6f22d06c0f05f1737 not found: ID does not exist" Nov 02 20:30:02.805105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.805080 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d7a94c-d41d-4122-8274-aec68f197f9f-kube-api-access-kj54k\" (UniqueName: \"kubernetes.io/projected/82d7a94c-d41d-4122-8274-aec68f197f9f-kube-api-access-kj54k\") pod \"82d7a94c-d41d-4122-8274-aec68f197f9f\" (UID: \"82d7a94c-d41d-4122-8274-aec68f197f9f\") " Nov 02 20:30:02.805498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.805266 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d7a94c-d41d-4122-8274-aec68f197f9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d7a94c-d41d-4122-8274-aec68f197f9f-plugins-config-volume\") pod \"82d7a94c-d41d-4122-8274-aec68f197f9f\" (UID: \"82d7a94c-d41d-4122-8274-aec68f197f9f\") " Nov 02 20:30:02.806979 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.806954 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d7a94c-d41d-4122-8274-aec68f197f9f-plugins-config-volume" pod "82d7a94c-d41d-4122-8274-aec68f197f9f" (UID: "82d7a94c-d41d-4122-8274-aec68f197f9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:02.808040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.807930 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d7a94c-d41d-4122-8274-aec68f197f9f-kube-api-access-kj54k" pod "82d7a94c-d41d-4122-8274-aec68f197f9f" (UID: "82d7a94c-d41d-4122-8274-aec68f197f9f"). InnerVolumeSpecName "kube-api-access-kj54k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:02.907781 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.907743 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj54k\" (UniqueName: \"kubernetes.io/projected/82d7a94c-d41d-4122-8274-aec68f197f9f-kube-api-access-kj54k\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:02.907781 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:02.907777 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d7a94c-d41d-4122-8274-aec68f197f9f-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:03.747224 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.747119 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" event={"ID":"c3621789-faf8-482e-bf9d-76caea94a3e2","Type":"ContainerStarted","Data":"40059a95094673e592b2d96111b7e3afdbd99703b28f661630f0fcd3e7923bb9"} Nov 02 20:30:03.749132 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.749106 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" event={"ID":"9e81e4f5-0154-4327-9de3-6690783b632a","Type":"ContainerStarted","Data":"7f99593087b5e5b5a1c5037d8aefc2b55d403010cc796746a0050ae041456949"} Nov 02 20:30:03.751061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.751039 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" event={"ID":"3fc569dc-8a4b-4774-9b77-a020e1d821a2","Type":"ContainerStarted","Data":"f7ba28adc323a732447867e7be452a6fa2a436106a6a7518660aefc5459f723a"} Nov 02 20:30:03.752908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.752878 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" event={"ID":"f8035ece-b691-4025-aab3-182741493aa0","Type":"ContainerStarted","Data":"8e07f3f46576b0938060e09ff7ac6c90b969ff2baf3f4c418a5231482f51a694"} Nov 02 20:30:03.754817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.754795 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" event={"ID":"dd0d8e94-5394-46a9-aceb-08532e885e00","Type":"ContainerStarted","Data":"695bce55bd44b878b97f907b845afd72768b243df4e5c75cbfaaf03bfc158af8"} Nov 02 20:30:03.756644 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.756620 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" event={"ID":"29bbab00-c103-433f-a3ec-90d6b1faec7b","Type":"ContainerStarted","Data":"4bc1e3a39f48b091946f1a6619e1d96c08279a5b672d93e05e690165b7f95762"} Nov 02 20:30:03.758416 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.758395 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" event={"ID":"d104368e-b2cc-4929-81b0-48a99a44c29f","Type":"ContainerStarted","Data":"a82a2fadef4f90ddf486b73b57e2c550392cadbf26abb6e407db940842986b06"} Nov 02 20:30:03.760302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.760267 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" event={"ID":"5eda0988-9984-43df-afa4-5ce94fa58c5a","Type":"ContainerStarted","Data":"df4435038bb0327300f088c66d35f62995e03ceb17379fed83bf231a1121fba0"} Nov 02 20:30:03.762114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.762096 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" event={"ID":"e254e9b3-e008-4c18-b6b1-786188414fa9","Type":"ContainerStarted","Data":"34be99dfb97831bf75ec18440303e0e29192679516b1bcd5c458e7f8c5bae1c2"} Nov 02 20:30:03.763019 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.763004 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl" Nov 02 20:30:03.908209 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.908171 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl"] Nov 02 20:30:03.938187 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:03.938149 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-429b45-d5f69c787-dshpl"] Nov 02 20:30:04.249062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.249001 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" podStartSLOduration=4.248987554 podStartE2EDuration="4.248987554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:04.246547194 +0000 UTC m=+2246.857199285" watchObservedRunningTime="2026-11-02 20:30:04.248987554 +0000 UTC m=+2246.859639706" Nov 02 20:30:04.568382 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.568328 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" podStartSLOduration=4.568314047 podStartE2EDuration="4.568314047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:04.564985588 +0000 UTC m=+2247.175637679" watchObservedRunningTime="2026-11-02 20:30:04.568314047 +0000 UTC m=+2247.178966137" Nov 02 20:30:04.772211 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.772181 5399 generic.go:356] "Generic (PLEG): container finished" podID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerID="40059a95094673e592b2d96111b7e3afdbd99703b28f661630f0fcd3e7923bb9" exitCode=60 Nov 02 20:30:04.772711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.772268 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" event={"ID":"c3621789-faf8-482e-bf9d-76caea94a3e2","Type":"ContainerDied","Data":"40059a95094673e592b2d96111b7e3afdbd99703b28f661630f0fcd3e7923bb9"} Nov 02 20:30:04.772711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.772346 5399 scope.go:122] "RemoveContainer" containerID="043873d70e64a3b14dec3b288adac170f4a64dc49689f0a9260ccf88b0d106e0" Nov 02 20:30:04.772711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.772635 5399 scope.go:122] "RemoveContainer" containerID="40059a95094673e592b2d96111b7e3afdbd99703b28f661630f0fcd3e7923bb9" Nov 02 20:30:04.773066 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:04.773043 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh_odh-ai-gateway-infra(c3621789-faf8-482e-bf9d-76caea94a3e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" Nov 02 20:30:04.774536 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.774513 5399 generic.go:356] "Generic (PLEG): container finished" podID="9e81e4f5-0154-4327-9de3-6690783b632a" containerID="7f99593087b5e5b5a1c5037d8aefc2b55d403010cc796746a0050ae041456949" exitCode=60 Nov 02 20:30:04.774616 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.774532 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" event={"ID":"9e81e4f5-0154-4327-9de3-6690783b632a","Type":"ContainerDied","Data":"7f99593087b5e5b5a1c5037d8aefc2b55d403010cc796746a0050ae041456949"} Nov 02 20:30:04.774850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.774835 5399 scope.go:122] "RemoveContainer" containerID="7f99593087b5e5b5a1c5037d8aefc2b55d403010cc796746a0050ae041456949" Nov 02 20:30:04.775208 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:04.775190 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws_odh-ai-gateway-infra(9e81e4f5-0154-4327-9de3-6690783b632a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" Nov 02 20:30:04.776588 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.776563 5399 generic.go:356] "Generic (PLEG): container finished" podID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerID="f7ba28adc323a732447867e7be452a6fa2a436106a6a7518660aefc5459f723a" exitCode=60 Nov 02 20:30:04.776703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.776648 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" event={"ID":"3fc569dc-8a4b-4774-9b77-a020e1d821a2","Type":"ContainerDied","Data":"f7ba28adc323a732447867e7be452a6fa2a436106a6a7518660aefc5459f723a"} Nov 02 20:30:04.777218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.777193 5399 scope.go:122] "RemoveContainer" containerID="f7ba28adc323a732447867e7be452a6fa2a436106a6a7518660aefc5459f723a" Nov 02 20:30:04.777628 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:04.777603 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj_odh-ai-gateway-infra(3fc569dc-8a4b-4774-9b77-a020e1d821a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" Nov 02 20:30:04.778902 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.778883 5399 generic.go:356] "Generic (PLEG): container finished" podID="f8035ece-b691-4025-aab3-182741493aa0" containerID="8e07f3f46576b0938060e09ff7ac6c90b969ff2baf3f4c418a5231482f51a694" exitCode=60 Nov 02 20:30:04.778984 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.778961 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" event={"ID":"f8035ece-b691-4025-aab3-182741493aa0","Type":"ContainerDied","Data":"8e07f3f46576b0938060e09ff7ac6c90b969ff2baf3f4c418a5231482f51a694"} Nov 02 20:30:04.779197 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.779181 5399 scope.go:122] "RemoveContainer" containerID="8e07f3f46576b0938060e09ff7ac6c90b969ff2baf3f4c418a5231482f51a694" Nov 02 20:30:04.779582 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:04.779558 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt_odh-ai-gateway-infra(f8035ece-b691-4025-aab3-182741493aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" podUID="f8035ece-b691-4025-aab3-182741493aa0" Nov 02 20:30:04.780903 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.780884 5399 generic.go:356] "Generic (PLEG): container finished" podID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerID="695bce55bd44b878b97f907b845afd72768b243df4e5c75cbfaaf03bfc158af8" exitCode=60 Nov 02 20:30:04.781002 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.780962 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" event={"ID":"dd0d8e94-5394-46a9-aceb-08532e885e00","Type":"ContainerDied","Data":"695bce55bd44b878b97f907b845afd72768b243df4e5c75cbfaaf03bfc158af8"} Nov 02 20:30:04.781210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.781190 5399 scope.go:122] "RemoveContainer" containerID="695bce55bd44b878b97f907b845afd72768b243df4e5c75cbfaaf03bfc158af8" Nov 02 20:30:04.781535 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:04.781511 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr_odh-ai-gateway-infra(dd0d8e94-5394-46a9-aceb-08532e885e00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" Nov 02 20:30:04.782997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.782976 5399 generic.go:356] "Generic (PLEG): container finished" podID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerID="4bc1e3a39f48b091946f1a6619e1d96c08279a5b672d93e05e690165b7f95762" exitCode=60 Nov 02 20:30:04.783081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.783040 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" event={"ID":"29bbab00-c103-433f-a3ec-90d6b1faec7b","Type":"ContainerDied","Data":"4bc1e3a39f48b091946f1a6619e1d96c08279a5b672d93e05e690165b7f95762"} Nov 02 20:30:04.783322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.783267 5399 scope.go:122] "RemoveContainer" containerID="4bc1e3a39f48b091946f1a6619e1d96c08279a5b672d93e05e690165b7f95762" Nov 02 20:30:04.783882 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:04.783580 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8_odh-ai-gateway-infra(29bbab00-c103-433f-a3ec-90d6b1faec7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" Nov 02 20:30:04.785731 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.785707 5399 generic.go:356] "Generic (PLEG): container finished" podID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerID="a82a2fadef4f90ddf486b73b57e2c550392cadbf26abb6e407db940842986b06" exitCode=60 Nov 02 20:30:04.785822 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.785731 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" event={"ID":"d104368e-b2cc-4929-81b0-48a99a44c29f","Type":"ContainerDied","Data":"a82a2fadef4f90ddf486b73b57e2c550392cadbf26abb6e407db940842986b06"} Nov 02 20:30:04.786026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.786010 5399 scope.go:122] "RemoveContainer" containerID="a82a2fadef4f90ddf486b73b57e2c550392cadbf26abb6e407db940842986b06" Nov 02 20:30:04.786414 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:04.786254 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9_odh-ai-gateway-infra(d104368e-b2cc-4929-81b0-48a99a44c29f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" Nov 02 20:30:04.788003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.787983 5399 generic.go:356] "Generic (PLEG): container finished" podID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerID="df4435038bb0327300f088c66d35f62995e03ceb17379fed83bf231a1121fba0" exitCode=60 Nov 02 20:30:04.788110 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.788087 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" event={"ID":"5eda0988-9984-43df-afa4-5ce94fa58c5a","Type":"ContainerDied","Data":"df4435038bb0327300f088c66d35f62995e03ceb17379fed83bf231a1121fba0"} Nov 02 20:30:04.788346 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.788328 5399 scope.go:122] "RemoveContainer" containerID="df4435038bb0327300f088c66d35f62995e03ceb17379fed83bf231a1121fba0" Nov 02 20:30:04.788586 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:04.788555 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8_odh-ai-gateway-infra(5eda0988-9984-43df-afa4-5ce94fa58c5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" Nov 02 20:30:04.789777 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.789759 5399 scope.go:122] "RemoveContainer" containerID="bf67a7f78b301632e4c6175cfb4b987b129e1faa3812aefac490ac6adf43dac6" Nov 02 20:30:04.790403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.790382 5399 generic.go:356] "Generic (PLEG): container finished" podID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerID="34be99dfb97831bf75ec18440303e0e29192679516b1bcd5c458e7f8c5bae1c2" exitCode=60 Nov 02 20:30:04.790502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.790410 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" event={"ID":"e254e9b3-e008-4c18-b6b1-786188414fa9","Type":"ContainerDied","Data":"34be99dfb97831bf75ec18440303e0e29192679516b1bcd5c458e7f8c5bae1c2"} Nov 02 20:30:04.790711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.790691 5399 scope.go:122] "RemoveContainer" containerID="34be99dfb97831bf75ec18440303e0e29192679516b1bcd5c458e7f8c5bae1c2" Nov 02 20:30:04.790963 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:04.790939 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n_odh-ai-gateway-infra(e254e9b3-e008-4c18-b6b1-786188414fa9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" Nov 02 20:30:04.805221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.805205 5399 scope.go:122] "RemoveContainer" containerID="8b0aee22c48c74b879996906d84a058755d0ee22af1dd39575de1af975ba0792" Nov 02 20:30:04.818946 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.818929 5399 scope.go:122] "RemoveContainer" containerID="f0ca82211ab20e9ea5d199313581334d546d5e7584855216a48ee836ab128a3f" Nov 02 20:30:04.833185 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.833163 5399 scope.go:122] "RemoveContainer" containerID="7f6e7bb4fb03671e84b6061a35692c12084abd371b502351af0e5935c77fee55" Nov 02 20:30:04.847064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.847041 5399 scope.go:122] "RemoveContainer" containerID="c0303b7e123114b9f12f42a0b124f098a37f348a48a1ac7b78ec0ae9e166c85d" Nov 02 20:30:04.860567 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.860545 5399 scope.go:122] "RemoveContainer" containerID="f0751ab85fba504d251aad16701dfeed986dad89498a145fa56c64a9b2aa4be2" Nov 02 20:30:04.874312 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.874257 5399 scope.go:122] "RemoveContainer" containerID="3c9d069d419f7cc3ca5a163b3afa52b219f437ffc675c97501025e9593e666a6" Nov 02 20:30:04.889197 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:04.889174 5399 scope.go:122] "RemoveContainer" containerID="3f702b83a245902e04ee4969b7549c23a5b0c335835564729acf3bd263a563f8" Nov 02 20:30:05.137810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.137706 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:30:05.510577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.510503 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d7a94c-d41d-4122-8274-aec68f197f9f" path="/var/lib/kubelet/pods/82d7a94c-d41d-4122-8274-aec68f197f9f/volumes" Nov 02 20:30:05.800679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.800600 5399 scope.go:122] "RemoveContainer" containerID="f7ba28adc323a732447867e7be452a6fa2a436106a6a7518660aefc5459f723a" Nov 02 20:30:05.801061 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:05.800956 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj_odh-ai-gateway-infra(3fc569dc-8a4b-4774-9b77-a020e1d821a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" Nov 02 20:30:05.802183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.802153 5399 scope.go:122] "RemoveContainer" containerID="8e07f3f46576b0938060e09ff7ac6c90b969ff2baf3f4c418a5231482f51a694" Nov 02 20:30:05.802538 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:05.802508 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt_odh-ai-gateway-infra(f8035ece-b691-4025-aab3-182741493aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" podUID="f8035ece-b691-4025-aab3-182741493aa0" Nov 02 20:30:05.803811 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.803794 5399 scope.go:122] "RemoveContainer" containerID="695bce55bd44b878b97f907b845afd72768b243df4e5c75cbfaaf03bfc158af8" Nov 02 20:30:05.804142 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:05.804120 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr_odh-ai-gateway-infra(dd0d8e94-5394-46a9-aceb-08532e885e00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" Nov 02 20:30:05.805422 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.805403 5399 scope.go:122] "RemoveContainer" containerID="4bc1e3a39f48b091946f1a6619e1d96c08279a5b672d93e05e690165b7f95762" Nov 02 20:30:05.805651 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:05.805635 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8_odh-ai-gateway-infra(29bbab00-c103-433f-a3ec-90d6b1faec7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" Nov 02 20:30:05.806904 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.806885 5399 scope.go:122] "RemoveContainer" containerID="a82a2fadef4f90ddf486b73b57e2c550392cadbf26abb6e407db940842986b06" Nov 02 20:30:05.807159 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:05.807131 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9_odh-ai-gateway-infra(d104368e-b2cc-4929-81b0-48a99a44c29f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" Nov 02 20:30:05.808393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.808377 5399 scope.go:122] "RemoveContainer" containerID="df4435038bb0327300f088c66d35f62995e03ceb17379fed83bf231a1121fba0" Nov 02 20:30:05.808610 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:05.808583 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8_odh-ai-gateway-infra(5eda0988-9984-43df-afa4-5ce94fa58c5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" Nov 02 20:30:05.809933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.809914 5399 scope.go:122] "RemoveContainer" containerID="34be99dfb97831bf75ec18440303e0e29192679516b1bcd5c458e7f8c5bae1c2" Nov 02 20:30:05.810173 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:05.810154 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n_odh-ai-gateway-infra(e254e9b3-e008-4c18-b6b1-786188414fa9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" Nov 02 20:30:05.811580 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.811563 5399 scope.go:122] "RemoveContainer" containerID="40059a95094673e592b2d96111b7e3afdbd99703b28f661630f0fcd3e7923bb9" Nov 02 20:30:05.811809 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:05.811787 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh_odh-ai-gateway-infra(c3621789-faf8-482e-bf9d-76caea94a3e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" Nov 02 20:30:05.813025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:05.813008 5399 scope.go:122] "RemoveContainer" containerID="7f99593087b5e5b5a1c5037d8aefc2b55d403010cc796746a0050ae041456949" Nov 02 20:30:05.813211 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:05.813195 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws_odh-ai-gateway-infra(9e81e4f5-0154-4327-9de3-6690783b632a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" Nov 02 20:30:11.025983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:11.025945 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j"] Nov 02 20:30:11.026560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:11.026148 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" containerID="cri-o://6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5" gracePeriod=30 Nov 02 20:30:15.501264 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:15.501229 5399 scope.go:122] "RemoveContainer" containerID="7f99593087b5e5b5a1c5037d8aefc2b55d403010cc796746a0050ae041456949" Nov 02 20:30:15.895999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:15.895957 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" event={"ID":"9e81e4f5-0154-4327-9de3-6690783b632a","Type":"ContainerStarted","Data":"202fe1510687708a1e90e0bce690beacfee2f02e8287bfb9327bce5f2aef9dea"} Nov 02 20:30:16.326227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.326201 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:30:16.412748 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.412669 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-credential-socket\") pod \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " Nov 02 20:30:16.412748 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.412744 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-podinfo\") pod \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " Nov 02 20:30:16.412991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.412771 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-envoy\") pod \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " Nov 02 20:30:16.412991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.412890 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-kube-api-access-qj4c6\" (UniqueName: \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-kube-api-access-qj4c6\") pod \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " Nov 02 20:30:16.413115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.413002 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-credential-socket" pod "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" (UID: "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:16.413115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.413041 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-token\" (UniqueName: \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-token\") pod \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " Nov 02 20:30:16.413212 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.413115 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-certs\") pod \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " Nov 02 20:30:16.413212 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.413160 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-socket\") pod \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " Nov 02 20:30:16.413376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.413257 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-data\") pod \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " Nov 02 20:30:16.413376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.413368 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istiod-ca-cert\") pod \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\" (UID: \"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5\") " Nov 02 20:30:16.413496 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.413422 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-certs" pod "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" (UID: "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:16.413577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.413540 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-data" pod "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" (UID: "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:16.413853 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.413827 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-socket" pod "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" (UID: "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:16.413853 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.413839 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istiod-ca-cert" pod "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" (UID: "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:16.414190 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.414166 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:16.414265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.414197 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:16.414265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.414215 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:16.414265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.414229 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:16.414265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.414243 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:16.415367 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.415328 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-envoy" pod "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" (UID: "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:16.415471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.415421 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-kube-api-access-qj4c6" pod "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" (UID: "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5"). InnerVolumeSpecName "kube-api-access-qj4c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:16.415471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.415423 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-podinfo" pod "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" (UID: "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:16.415699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.415666 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-token" pod "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" (UID: "7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:16.501475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.501446 5399 scope.go:122] "RemoveContainer" containerID="4bc1e3a39f48b091946f1a6619e1d96c08279a5b672d93e05e690165b7f95762" Nov 02 20:30:16.502057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.502039 5399 scope.go:122] "RemoveContainer" containerID="f7ba28adc323a732447867e7be452a6fa2a436106a6a7518660aefc5459f723a" Nov 02 20:30:16.516336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.516308 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:16.516462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.516340 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:16.516462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.516355 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:16.516462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.516369 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj4c6\" (UniqueName: \"kubernetes.io/projected/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5-kube-api-access-qj4c6\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:16.906093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.906012 5399 generic.go:356] "Generic (PLEG): container finished" podID="9e81e4f5-0154-4327-9de3-6690783b632a" containerID="202fe1510687708a1e90e0bce690beacfee2f02e8287bfb9327bce5f2aef9dea" exitCode=60 Nov 02 20:30:16.906093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.906069 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" event={"ID":"9e81e4f5-0154-4327-9de3-6690783b632a","Type":"ContainerDied","Data":"202fe1510687708a1e90e0bce690beacfee2f02e8287bfb9327bce5f2aef9dea"} Nov 02 20:30:16.906399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.906133 5399 scope.go:122] "RemoveContainer" containerID="7f99593087b5e5b5a1c5037d8aefc2b55d403010cc796746a0050ae041456949" Nov 02 20:30:16.906609 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.906585 5399 scope.go:122] "RemoveContainer" containerID="202fe1510687708a1e90e0bce690beacfee2f02e8287bfb9327bce5f2aef9dea" Nov 02 20:30:16.907143 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:16.907017 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws_odh-ai-gateway-infra(9e81e4f5-0154-4327-9de3-6690783b632a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" Nov 02 20:30:16.908494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.908473 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" event={"ID":"3fc569dc-8a4b-4774-9b77-a020e1d821a2","Type":"ContainerStarted","Data":"3c1b2b0136bbc01d17e83ff0300bd6a670ad845778797acb50d18bc62ad68e12"} Nov 02 20:30:16.910369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.910343 5399 generic.go:356] "Generic (PLEG): container finished" podID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerID="6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5" exitCode=0 Nov 02 20:30:16.910459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.910389 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" Nov 02 20:30:16.910459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.910402 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" event={"ID":"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5","Type":"ContainerDied","Data":"6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5"} Nov 02 20:30:16.910540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.910459 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j" event={"ID":"7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5","Type":"ContainerDied","Data":"0588e20adfae2c58c567ebff5e8096e0a3a18751cee30beab17b75f38562dfe6"} Nov 02 20:30:16.920504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.920486 5399 scope.go:122] "RemoveContainer" containerID="6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5" Nov 02 20:30:16.931435 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.931414 5399 scope.go:122] "RemoveContainer" containerID="01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2" Nov 02 20:30:16.944886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.944865 5399 scope.go:122] "RemoveContainer" containerID="6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5" Nov 02 20:30:16.945353 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:16.945270 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5\": container with ID starting with 6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5 not found: ID does not exist" containerID="6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5" Nov 02 20:30:16.945471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.945362 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5"} err="failed to get container status \"6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5\": rpc error: code = NotFound desc = could not find container \"6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5\": container with ID starting with 6ab93d757cab7d6a0ab0835af64a687ec1a9b9044828f589d6ccedcaa31cbce5 not found: ID does not exist" Nov 02 20:30:16.945471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.945382 5399 scope.go:122] "RemoveContainer" containerID="01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2" Nov 02 20:30:16.946492 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:16.946462 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2\": container with ID starting with 01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2 not found: ID does not exist" containerID="01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2" Nov 02 20:30:16.946593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:16.946498 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2"} err="failed to get container status \"01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2\": rpc error: code = NotFound desc = could not find container \"01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2\": container with ID starting with 01670c6328e5b2c5011c0c158f481430b38199fdeff25e46e8911072c3ee23e2 not found: ID does not exist" Nov 02 20:30:17.014824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.014743 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws"] Nov 02 20:30:17.127834 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.127787 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j"] Nov 02 20:30:17.131701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.131669 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-eb060a-openshift-default-b8bbb8f9c-qb89j"] Nov 02 20:30:17.513343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.513309 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" path="/var/lib/kubelet/pods/7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5/volumes" Nov 02 20:30:17.514408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.514383 5399 scope.go:122] "RemoveContainer" containerID="695bce55bd44b878b97f907b845afd72768b243df4e5c75cbfaaf03bfc158af8" Nov 02 20:30:17.514593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.514575 5399 scope.go:122] "RemoveContainer" containerID="34be99dfb97831bf75ec18440303e0e29192679516b1bcd5c458e7f8c5bae1c2" Nov 02 20:30:17.514764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.514744 5399 scope.go:122] "RemoveContainer" containerID="8e07f3f46576b0938060e09ff7ac6c90b969ff2baf3f4c418a5231482f51a694" Nov 02 20:30:17.921918 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.921887 5399 generic.go:356] "Generic (PLEG): container finished" podID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerID="3c1b2b0136bbc01d17e83ff0300bd6a670ad845778797acb50d18bc62ad68e12" exitCode=60 Nov 02 20:30:17.922026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.921963 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" event={"ID":"3fc569dc-8a4b-4774-9b77-a020e1d821a2","Type":"ContainerDied","Data":"3c1b2b0136bbc01d17e83ff0300bd6a670ad845778797acb50d18bc62ad68e12"} Nov 02 20:30:17.922106 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.922042 5399 scope.go:122] "RemoveContainer" containerID="f7ba28adc323a732447867e7be452a6fa2a436106a6a7518660aefc5459f723a" Nov 02 20:30:17.922515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.922490 5399 scope.go:122] "RemoveContainer" containerID="3c1b2b0136bbc01d17e83ff0300bd6a670ad845778797acb50d18bc62ad68e12" Nov 02 20:30:17.922897 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:17.922867 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj_odh-ai-gateway-infra(3fc569dc-8a4b-4774-9b77-a020e1d821a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" Nov 02 20:30:17.924112 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.924079 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" event={"ID":"f8035ece-b691-4025-aab3-182741493aa0","Type":"ContainerStarted","Data":"4f5cea61a8683e02ec5418f426f9b674a2baa97946841fa1135e9d5d6547fae9"} Nov 02 20:30:17.926420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.926395 5399 generic.go:356] "Generic (PLEG): container finished" podID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerID="59660fd54f57748dfa0f9e8df251c3771004db54422ec87c7ffd76ab6caacf9f" exitCode=60 Nov 02 20:30:17.926508 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.926469 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" event={"ID":"29bbab00-c103-433f-a3ec-90d6b1faec7b","Type":"ContainerDied","Data":"59660fd54f57748dfa0f9e8df251c3771004db54422ec87c7ffd76ab6caacf9f"} Nov 02 20:30:17.926955 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.926938 5399 scope.go:122] "RemoveContainer" containerID="59660fd54f57748dfa0f9e8df251c3771004db54422ec87c7ffd76ab6caacf9f" Nov 02 20:30:17.927252 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:17.927227 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8_odh-ai-gateway-infra(29bbab00-c103-433f-a3ec-90d6b1faec7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" Nov 02 20:30:17.950124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:17.950101 5399 scope.go:122] "RemoveContainer" containerID="4bc1e3a39f48b091946f1a6619e1d96c08279a5b672d93e05e690165b7f95762" Nov 02 20:30:18.087007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.086982 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" Nov 02 20:30:18.245642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.244523 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e81e4f5-0154-4327-9de3-6690783b632a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e81e4f5-0154-4327-9de3-6690783b632a-maas-api-service-ca\") pod \"9e81e4f5-0154-4327-9de3-6690783b632a\" (UID: \"9e81e4f5-0154-4327-9de3-6690783b632a\") " Nov 02 20:30:18.245642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.244657 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e81e4f5-0154-4327-9de3-6690783b632a-kube-api-access-ghrkh\" (UniqueName: \"kubernetes.io/projected/9e81e4f5-0154-4327-9de3-6690783b632a-kube-api-access-ghrkh\") pod \"9e81e4f5-0154-4327-9de3-6690783b632a\" (UID: \"9e81e4f5-0154-4327-9de3-6690783b632a\") " Nov 02 20:30:18.245642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.245046 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e81e4f5-0154-4327-9de3-6690783b632a-maas-api-service-ca" pod "9e81e4f5-0154-4327-9de3-6690783b632a" (UID: "9e81e4f5-0154-4327-9de3-6690783b632a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:18.245642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.245595 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e81e4f5-0154-4327-9de3-6690783b632a-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:18.248220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.248155 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj"] Nov 02 20:30:18.248400 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.248322 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e81e4f5-0154-4327-9de3-6690783b632a-kube-api-access-ghrkh" pod "9e81e4f5-0154-4327-9de3-6690783b632a" (UID: "9e81e4f5-0154-4327-9de3-6690783b632a"). InnerVolumeSpecName "kube-api-access-ghrkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:18.248511 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:18.248480 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:30:18.346914 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.346875 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghrkh\" (UniqueName: \"kubernetes.io/projected/9e81e4f5-0154-4327-9de3-6690783b632a-kube-api-access-ghrkh\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:18.512347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.502057 5399 scope.go:122] "RemoveContainer" containerID="df4435038bb0327300f088c66d35f62995e03ceb17379fed83bf231a1121fba0" Nov 02 20:30:18.512347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.502799 5399 scope.go:122] "RemoveContainer" containerID="40059a95094673e592b2d96111b7e3afdbd99703b28f661630f0fcd3e7923bb9" Nov 02 20:30:18.546732 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.545529 5399 scope.go:122] "RemoveContainer" containerID="a82a2fadef4f90ddf486b73b57e2c550392cadbf26abb6e407db940842986b06" Nov 02 20:30:18.970305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.969925 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" event={"ID":"9e81e4f5-0154-4327-9de3-6690783b632a","Type":"ContainerDied","Data":"94ae7753ebfc63a9d5ea2f4778a9becd9b9642753c78e434c257ea04259924dd"} Nov 02 20:30:18.970305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.970070 5399 scope.go:122] "RemoveContainer" containerID="202fe1510687708a1e90e0bce690beacfee2f02e8287bfb9327bce5f2aef9dea" Nov 02 20:30:18.970305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.970194 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws" Nov 02 20:30:18.989408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.987478 5399 generic.go:356] "Generic (PLEG): container finished" podID="f8035ece-b691-4025-aab3-182741493aa0" containerID="4f5cea61a8683e02ec5418f426f9b674a2baa97946841fa1135e9d5d6547fae9" exitCode=60 Nov 02 20:30:18.989408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.987589 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" event={"ID":"f8035ece-b691-4025-aab3-182741493aa0","Type":"ContainerDied","Data":"4f5cea61a8683e02ec5418f426f9b674a2baa97946841fa1135e9d5d6547fae9"} Nov 02 20:30:18.989408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.988059 5399 scope.go:122] "RemoveContainer" containerID="4f5cea61a8683e02ec5418f426f9b674a2baa97946841fa1135e9d5d6547fae9" Nov 02 20:30:18.989408 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:18.988479 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt_odh-ai-gateway-infra(f8035ece-b691-4025-aab3-182741493aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" podUID="f8035ece-b691-4025-aab3-182741493aa0" Nov 02 20:30:19.000244 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.996908 5399 generic.go:356] "Generic (PLEG): container finished" podID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerID="5fea8fabfdbc365d1a78a914f10066742bd4446e90515495a8f0c10bc9d02664" exitCode=60 Nov 02 20:30:19.000244 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.996994 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" event={"ID":"dd0d8e94-5394-46a9-aceb-08532e885e00","Type":"ContainerDied","Data":"5fea8fabfdbc365d1a78a914f10066742bd4446e90515495a8f0c10bc9d02664"} Nov 02 20:30:19.000244 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:18.997451 5399 scope.go:122] "RemoveContainer" containerID="5fea8fabfdbc365d1a78a914f10066742bd4446e90515495a8f0c10bc9d02664" Nov 02 20:30:19.000244 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:18.997913 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr_odh-ai-gateway-infra(dd0d8e94-5394-46a9-aceb-08532e885e00)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" Nov 02 20:30:19.003206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.002202 5399 generic.go:356] "Generic (PLEG): container finished" podID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerID="a754ce39cd9ce066ac463cdbb651d8a12ef474ed98189df74e7158dbe5c46a37" exitCode=60 Nov 02 20:30:19.003206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.002287 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" event={"ID":"e254e9b3-e008-4c18-b6b1-786188414fa9","Type":"ContainerDied","Data":"a754ce39cd9ce066ac463cdbb651d8a12ef474ed98189df74e7158dbe5c46a37"} Nov 02 20:30:19.003206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.002721 5399 scope.go:122] "RemoveContainer" containerID="a754ce39cd9ce066ac463cdbb651d8a12ef474ed98189df74e7158dbe5c46a37" Nov 02 20:30:19.003206 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:19.003131 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n_odh-ai-gateway-infra(e254e9b3-e008-4c18-b6b1-786188414fa9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" Nov 02 20:30:19.017874 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.017773 5399 scope.go:122] "RemoveContainer" containerID="8e07f3f46576b0938060e09ff7ac6c90b969ff2baf3f4c418a5231482f51a694" Nov 02 20:30:19.039395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.039372 5399 scope.go:122] "RemoveContainer" containerID="695bce55bd44b878b97f907b845afd72768b243df4e5c75cbfaaf03bfc158af8" Nov 02 20:30:19.054149 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.054124 5399 scope.go:122] "RemoveContainer" containerID="34be99dfb97831bf75ec18440303e0e29192679516b1bcd5c458e7f8c5bae1c2" Nov 02 20:30:19.193025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.192997 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" Nov 02 20:30:19.254578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.254542 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8"] Nov 02 20:30:19.288357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.288324 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc569dc-8a4b-4774-9b77-a020e1d821a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc569dc-8a4b-4774-9b77-a020e1d821a2-maas-api-service-ca\") pod \"3fc569dc-8a4b-4774-9b77-a020e1d821a2\" (UID: \"3fc569dc-8a4b-4774-9b77-a020e1d821a2\") " Nov 02 20:30:19.288590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.288432 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc569dc-8a4b-4774-9b77-a020e1d821a2-kube-api-access-mg6dh\" (UniqueName: \"kubernetes.io/projected/3fc569dc-8a4b-4774-9b77-a020e1d821a2-kube-api-access-mg6dh\") pod \"3fc569dc-8a4b-4774-9b77-a020e1d821a2\" (UID: \"3fc569dc-8a4b-4774-9b77-a020e1d821a2\") " Nov 02 20:30:19.288809 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.288782 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc569dc-8a4b-4774-9b77-a020e1d821a2-maas-api-service-ca" pod "3fc569dc-8a4b-4774-9b77-a020e1d821a2" (UID: "3fc569dc-8a4b-4774-9b77-a020e1d821a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:19.289086 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.289068 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc569dc-8a4b-4774-9b77-a020e1d821a2-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:19.292384 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.292331 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc569dc-8a4b-4774-9b77-a020e1d821a2-kube-api-access-mg6dh" pod "3fc569dc-8a4b-4774-9b77-a020e1d821a2" (UID: "3fc569dc-8a4b-4774-9b77-a020e1d821a2"). InnerVolumeSpecName "kube-api-access-mg6dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:19.362362 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.362326 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt"] Nov 02 20:30:19.390950 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.390918 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg6dh\" (UniqueName: \"kubernetes.io/projected/3fc569dc-8a4b-4774-9b77-a020e1d821a2-kube-api-access-mg6dh\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:19.401258 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.401235 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" Nov 02 20:30:19.491631 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.491548 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bbab00-c103-433f-a3ec-90d6b1faec7b-kube-api-access-gcw9s\" (UniqueName: \"kubernetes.io/projected/29bbab00-c103-433f-a3ec-90d6b1faec7b-kube-api-access-gcw9s\") pod \"29bbab00-c103-433f-a3ec-90d6b1faec7b\" (UID: \"29bbab00-c103-433f-a3ec-90d6b1faec7b\") " Nov 02 20:30:19.491904 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.491882 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29bbab00-c103-433f-a3ec-90d6b1faec7b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bbab00-c103-433f-a3ec-90d6b1faec7b-maas-api-service-ca\") pod \"29bbab00-c103-433f-a3ec-90d6b1faec7b\" (UID: \"29bbab00-c103-433f-a3ec-90d6b1faec7b\") " Nov 02 20:30:19.492327 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.492265 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bbab00-c103-433f-a3ec-90d6b1faec7b-maas-api-service-ca" pod "29bbab00-c103-433f-a3ec-90d6b1faec7b" (UID: "29bbab00-c103-433f-a3ec-90d6b1faec7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:19.492553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.492538 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bbab00-c103-433f-a3ec-90d6b1faec7b-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:19.493641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.493621 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bbab00-c103-433f-a3ec-90d6b1faec7b-kube-api-access-gcw9s" pod "29bbab00-c103-433f-a3ec-90d6b1faec7b" (UID: "29bbab00-c103-433f-a3ec-90d6b1faec7b"). InnerVolumeSpecName "kube-api-access-gcw9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:19.553002 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.552968 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws"] Nov 02 20:30:19.557176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.557150 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c76017-29894190-7z7ws"] Nov 02 20:30:19.594220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:19.594190 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcw9s\" (UniqueName: \"kubernetes.io/projected/29bbab00-c103-433f-a3ec-90d6b1faec7b-kube-api-access-gcw9s\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:20.016515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.016424 5399 generic.go:356] "Generic (PLEG): container finished" podID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerID="553502bf4b1ce71d7cbd9a588b69ebb82748d558c070674eced84e2b75db74ae" exitCode=60 Nov 02 20:30:20.016686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.016505 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" event={"ID":"c3621789-faf8-482e-bf9d-76caea94a3e2","Type":"ContainerDied","Data":"553502bf4b1ce71d7cbd9a588b69ebb82748d558c070674eced84e2b75db74ae"} Nov 02 20:30:20.016686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.016582 5399 scope.go:122] "RemoveContainer" containerID="40059a95094673e592b2d96111b7e3afdbd99703b28f661630f0fcd3e7923bb9" Nov 02 20:30:20.017117 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.017092 5399 scope.go:122] "RemoveContainer" containerID="553502bf4b1ce71d7cbd9a588b69ebb82748d558c070674eced84e2b75db74ae" Nov 02 20:30:20.017560 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:20.017536 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh_odh-ai-gateway-infra(c3621789-faf8-482e-bf9d-76caea94a3e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" Nov 02 20:30:20.020428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.020404 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" Nov 02 20:30:20.020428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.020424 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj" event={"ID":"3fc569dc-8a4b-4774-9b77-a020e1d821a2","Type":"ContainerDied","Data":"644b95c42750d56920453eb237a8c55cef473464c70f2f47d855e39755249f66"} Nov 02 20:30:20.027198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.027170 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" event={"ID":"29bbab00-c103-433f-a3ec-90d6b1faec7b","Type":"ContainerDied","Data":"a24ee7c184e136076b8fd48eb5bcb2b7c266828dcb4fa9aee8c8971f0fc62c77"} Nov 02 20:30:20.027343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.027325 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8" Nov 02 20:30:20.031448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.031421 5399 generic.go:356] "Generic (PLEG): container finished" podID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerID="86df052e578a210aa35ed30309a7bd447b4d15cb820cefac160e443758f9f833" exitCode=60 Nov 02 20:30:20.031542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.031522 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" event={"ID":"d104368e-b2cc-4929-81b0-48a99a44c29f","Type":"ContainerDied","Data":"86df052e578a210aa35ed30309a7bd447b4d15cb820cefac160e443758f9f833"} Nov 02 20:30:20.031981 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.031956 5399 scope.go:122] "RemoveContainer" containerID="86df052e578a210aa35ed30309a7bd447b4d15cb820cefac160e443758f9f833" Nov 02 20:30:20.032356 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:20.032326 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9_odh-ai-gateway-infra(d104368e-b2cc-4929-81b0-48a99a44c29f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" Nov 02 20:30:20.034594 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.034567 5399 generic.go:356] "Generic (PLEG): container finished" podID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerID="50195bd172165269f2369de8c4ae3f68e0bef6ecc56fecd87198815cd2748f89" exitCode=60 Nov 02 20:30:20.034690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.034609 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" event={"ID":"5eda0988-9984-43df-afa4-5ce94fa58c5a","Type":"ContainerDied","Data":"50195bd172165269f2369de8c4ae3f68e0bef6ecc56fecd87198815cd2748f89"} Nov 02 20:30:20.035027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.035008 5399 scope.go:122] "RemoveContainer" containerID="50195bd172165269f2369de8c4ae3f68e0bef6ecc56fecd87198815cd2748f89" Nov 02 20:30:20.035397 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:20.035371 5399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8_odh-ai-gateway-infra(5eda0988-9984-43df-afa4-5ce94fa58c5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" Nov 02 20:30:20.035497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.035481 5399 scope.go:122] "RemoveContainer" containerID="3c1b2b0136bbc01d17e83ff0300bd6a670ad845778797acb50d18bc62ad68e12" Nov 02 20:30:20.056580 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.056558 5399 scope.go:122] "RemoveContainer" containerID="59660fd54f57748dfa0f9e8df251c3771004db54422ec87c7ffd76ab6caacf9f" Nov 02 20:30:20.069560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.068969 5399 scope.go:122] "RemoveContainer" containerID="a82a2fadef4f90ddf486b73b57e2c550392cadbf26abb6e407db940842986b06" Nov 02 20:30:20.085332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.085307 5399 scope.go:122] "RemoveContainer" containerID="df4435038bb0327300f088c66d35f62995e03ceb17379fed83bf231a1121fba0" Nov 02 20:30:20.190815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.190789 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" Nov 02 20:30:20.229842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.229808 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8"] Nov 02 20:30:20.238720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.238692 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-40425f-29894190-4wsm8"] Nov 02 20:30:20.305682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.305658 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8035ece-b691-4025-aab3-182741493aa0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8035ece-b691-4025-aab3-182741493aa0-maas-api-service-ca\") pod \"f8035ece-b691-4025-aab3-182741493aa0\" (UID: \"f8035ece-b691-4025-aab3-182741493aa0\") " Nov 02 20:30:20.305838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.305725 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8035ece-b691-4025-aab3-182741493aa0-kube-api-access-9d8qb\" (UniqueName: \"kubernetes.io/projected/f8035ece-b691-4025-aab3-182741493aa0-kube-api-access-9d8qb\") pod \"f8035ece-b691-4025-aab3-182741493aa0\" (UID: \"f8035ece-b691-4025-aab3-182741493aa0\") " Nov 02 20:30:20.306124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.306099 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8035ece-b691-4025-aab3-182741493aa0-maas-api-service-ca" pod "f8035ece-b691-4025-aab3-182741493aa0" (UID: "f8035ece-b691-4025-aab3-182741493aa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:20.306465 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.306444 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8035ece-b691-4025-aab3-182741493aa0-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:20.308028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.308003 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8035ece-b691-4025-aab3-182741493aa0-kube-api-access-9d8qb" pod "f8035ece-b691-4025-aab3-182741493aa0" (UID: "f8035ece-b691-4025-aab3-182741493aa0"). InnerVolumeSpecName "kube-api-access-9d8qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:20.350604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.350574 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n"] Nov 02 20:30:20.408240 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.408208 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d8qb\" (UniqueName: \"kubernetes.io/projected/f8035ece-b691-4025-aab3-182741493aa0-kube-api-access-9d8qb\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:20.462181 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.462140 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr"] Nov 02 20:30:20.496594 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.496490 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" Nov 02 20:30:20.559597 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.558769 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj"] Nov 02 20:30:20.564219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.564188 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9fcedd-29894190-t6xgj"] Nov 02 20:30:20.611075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.611033 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e254e9b3-e008-4c18-b6b1-786188414fa9-kube-api-access-7mfv8\" (UniqueName: \"kubernetes.io/projected/e254e9b3-e008-4c18-b6b1-786188414fa9-kube-api-access-7mfv8\") pod \"e254e9b3-e008-4c18-b6b1-786188414fa9\" (UID: \"e254e9b3-e008-4c18-b6b1-786188414fa9\") " Nov 02 20:30:20.611520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.611491 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e254e9b3-e008-4c18-b6b1-786188414fa9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e254e9b3-e008-4c18-b6b1-786188414fa9-maas-api-service-ca\") pod \"e254e9b3-e008-4c18-b6b1-786188414fa9\" (UID: \"e254e9b3-e008-4c18-b6b1-786188414fa9\") " Nov 02 20:30:20.613258 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.613219 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e254e9b3-e008-4c18-b6b1-786188414fa9-maas-api-service-ca" pod "e254e9b3-e008-4c18-b6b1-786188414fa9" (UID: "e254e9b3-e008-4c18-b6b1-786188414fa9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:20.615948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.615911 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e254e9b3-e008-4c18-b6b1-786188414fa9-kube-api-access-7mfv8" pod "e254e9b3-e008-4c18-b6b1-786188414fa9" (UID: "e254e9b3-e008-4c18-b6b1-786188414fa9"). InnerVolumeSpecName "kube-api-access-7mfv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:20.655757 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.655730 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" Nov 02 20:30:20.713957 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.713924 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mfv8\" (UniqueName: \"kubernetes.io/projected/e254e9b3-e008-4c18-b6b1-786188414fa9-kube-api-access-7mfv8\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:20.713957 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.713954 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e254e9b3-e008-4c18-b6b1-786188414fa9-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:20.815639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.815537 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0d8e94-5394-46a9-aceb-08532e885e00-kube-api-access-qm6mb\" (UniqueName: \"kubernetes.io/projected/dd0d8e94-5394-46a9-aceb-08532e885e00-kube-api-access-qm6mb\") pod \"dd0d8e94-5394-46a9-aceb-08532e885e00\" (UID: \"dd0d8e94-5394-46a9-aceb-08532e885e00\") " Nov 02 20:30:20.815800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.815762 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0d8e94-5394-46a9-aceb-08532e885e00-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0d8e94-5394-46a9-aceb-08532e885e00-maas-api-service-ca\") pod \"dd0d8e94-5394-46a9-aceb-08532e885e00\" (UID: \"dd0d8e94-5394-46a9-aceb-08532e885e00\") " Nov 02 20:30:20.816417 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.816382 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0d8e94-5394-46a9-aceb-08532e885e00-maas-api-service-ca" pod "dd0d8e94-5394-46a9-aceb-08532e885e00" (UID: "dd0d8e94-5394-46a9-aceb-08532e885e00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:20.818106 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.818063 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0d8e94-5394-46a9-aceb-08532e885e00-kube-api-access-qm6mb" pod "dd0d8e94-5394-46a9-aceb-08532e885e00" (UID: "dd0d8e94-5394-46a9-aceb-08532e885e00"). InnerVolumeSpecName "kube-api-access-qm6mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:20.918201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.918157 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm6mb\" (UniqueName: \"kubernetes.io/projected/dd0d8e94-5394-46a9-aceb-08532e885e00-kube-api-access-qm6mb\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:20.918201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:20.918201 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0d8e94-5394-46a9-aceb-08532e885e00-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.048925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.048885 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" Nov 02 20:30:21.049123 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.048922 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n" event={"ID":"e254e9b3-e008-4c18-b6b1-786188414fa9","Type":"ContainerDied","Data":"f6584240fbf2df79cef8202db2f546b29cdc4f4d379f6d737ac4bad156f01d4f"} Nov 02 20:30:21.049123 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.049027 5399 scope.go:122] "RemoveContainer" containerID="a754ce39cd9ce066ac463cdbb651d8a12ef474ed98189df74e7158dbe5c46a37" Nov 02 20:30:21.054141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.054118 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" Nov 02 20:30:21.054641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.054136 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt" event={"ID":"f8035ece-b691-4025-aab3-182741493aa0","Type":"ContainerDied","Data":"5090dddb7c60cc389c9a5dbfa2e345425791d10b47b684a4e379effe99d6c991"} Nov 02 20:30:21.055796 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.055773 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" Nov 02 20:30:21.055880 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.055788 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr" event={"ID":"dd0d8e94-5394-46a9-aceb-08532e885e00","Type":"ContainerDied","Data":"42d4ec103c608b8265919a1c361645a91b59abeb1a3d14835e23a24f43eac341"} Nov 02 20:30:21.063455 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.063437 5399 scope.go:122] "RemoveContainer" containerID="4f5cea61a8683e02ec5418f426f9b674a2baa97946841fa1135e9d5d6547fae9" Nov 02 20:30:21.074203 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.074181 5399 scope.go:122] "RemoveContainer" containerID="5fea8fabfdbc365d1a78a914f10066742bd4446e90515495a8f0c10bc9d02664" Nov 02 20:30:21.136995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.136960 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh"] Nov 02 20:30:21.205881 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.205484 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt"] Nov 02 20:30:21.215126 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.213946 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2052e3-29894190-ljldt"] Nov 02 20:30:21.313095 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.313061 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n"] Nov 02 20:30:21.319057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.318994 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4dc8ca-29894190-hgs6n"] Nov 02 20:30:21.336745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.336724 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" Nov 02 20:30:21.353583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.353558 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8"] Nov 02 20:30:21.429905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.429874 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3621789-faf8-482e-bf9d-76caea94a3e2-kube-api-access-8s688\" (UniqueName: \"kubernetes.io/projected/c3621789-faf8-482e-bf9d-76caea94a3e2-kube-api-access-8s688\") pod \"c3621789-faf8-482e-bf9d-76caea94a3e2\" (UID: \"c3621789-faf8-482e-bf9d-76caea94a3e2\") " Nov 02 20:30:21.430096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.430015 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3621789-faf8-482e-bf9d-76caea94a3e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3621789-faf8-482e-bf9d-76caea94a3e2-maas-api-service-ca\") pod \"c3621789-faf8-482e-bf9d-76caea94a3e2\" (UID: \"c3621789-faf8-482e-bf9d-76caea94a3e2\") " Nov 02 20:30:21.431038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.430981 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3621789-faf8-482e-bf9d-76caea94a3e2-maas-api-service-ca" pod "c3621789-faf8-482e-bf9d-76caea94a3e2" (UID: "c3621789-faf8-482e-bf9d-76caea94a3e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:21.433841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.433806 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3621789-faf8-482e-bf9d-76caea94a3e2-kube-api-access-8s688" pod "c3621789-faf8-482e-bf9d-76caea94a3e2" (UID: "c3621789-faf8-482e-bf9d-76caea94a3e2"). InnerVolumeSpecName "kube-api-access-8s688". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:21.436470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.436438 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr"] Nov 02 20:30:21.442434 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.442410 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-8c5e8e-29894190-w7cbr"] Nov 02 20:30:21.488087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.488052 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" Nov 02 20:30:21.508774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.508750 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" path="/var/lib/kubelet/pods/29bbab00-c103-433f-a3ec-90d6b1faec7b/volumes" Nov 02 20:30:21.509222 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.509204 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" path="/var/lib/kubelet/pods/3fc569dc-8a4b-4774-9b77-a020e1d821a2/volumes" Nov 02 20:30:21.509644 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.509628 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" path="/var/lib/kubelet/pods/9e81e4f5-0154-4327-9de3-6690783b632a/volumes" Nov 02 20:30:21.510363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.510345 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" path="/var/lib/kubelet/pods/dd0d8e94-5394-46a9-aceb-08532e885e00/volumes" Nov 02 20:30:21.510837 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.510819 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" path="/var/lib/kubelet/pods/e254e9b3-e008-4c18-b6b1-786188414fa9/volumes" Nov 02 20:30:21.511244 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.511229 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8035ece-b691-4025-aab3-182741493aa0" path="/var/lib/kubelet/pods/f8035ece-b691-4025-aab3-182741493aa0/volumes" Nov 02 20:30:21.530999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.530973 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eda0988-9984-43df-afa4-5ce94fa58c5a-kube-api-access-plhjw\" (UniqueName: \"kubernetes.io/projected/5eda0988-9984-43df-afa4-5ce94fa58c5a-kube-api-access-plhjw\") pod \"5eda0988-9984-43df-afa4-5ce94fa58c5a\" (UID: \"5eda0988-9984-43df-afa4-5ce94fa58c5a\") " Nov 02 20:30:21.531155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.531141 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eda0988-9984-43df-afa4-5ce94fa58c5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda0988-9984-43df-afa4-5ce94fa58c5a-maas-api-service-ca\") pod \"5eda0988-9984-43df-afa4-5ce94fa58c5a\" (UID: \"5eda0988-9984-43df-afa4-5ce94fa58c5a\") " Nov 02 20:30:21.531513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.531487 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eda0988-9984-43df-afa4-5ce94fa58c5a-maas-api-service-ca" pod "5eda0988-9984-43df-afa4-5ce94fa58c5a" (UID: "5eda0988-9984-43df-afa4-5ce94fa58c5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:21.531838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.531814 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s688\" (UniqueName: \"kubernetes.io/projected/c3621789-faf8-482e-bf9d-76caea94a3e2-kube-api-access-8s688\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.531838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.531835 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3621789-faf8-482e-bf9d-76caea94a3e2-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.532012 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.531845 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda0988-9984-43df-afa4-5ce94fa58c5a-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.533405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.533384 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eda0988-9984-43df-afa4-5ce94fa58c5a-kube-api-access-plhjw" pod "5eda0988-9984-43df-afa4-5ce94fa58c5a" (UID: "5eda0988-9984-43df-afa4-5ce94fa58c5a"). InnerVolumeSpecName "kube-api-access-plhjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:21.633178 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.633093 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plhjw\" (UniqueName: \"kubernetes.io/projected/5eda0988-9984-43df-afa4-5ce94fa58c5a-kube-api-access-plhjw\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.678037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.678002 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9"] Nov 02 20:30:21.847061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.847027 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" Nov 02 20:30:21.936448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.936364 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d104368e-b2cc-4929-81b0-48a99a44c29f-kube-api-access-qjzsh\" (UniqueName: \"kubernetes.io/projected/d104368e-b2cc-4929-81b0-48a99a44c29f-kube-api-access-qjzsh\") pod \"d104368e-b2cc-4929-81b0-48a99a44c29f\" (UID: \"d104368e-b2cc-4929-81b0-48a99a44c29f\") " Nov 02 20:30:21.936591 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.936566 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d104368e-b2cc-4929-81b0-48a99a44c29f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d104368e-b2cc-4929-81b0-48a99a44c29f-maas-api-service-ca\") pod \"d104368e-b2cc-4929-81b0-48a99a44c29f\" (UID: \"d104368e-b2cc-4929-81b0-48a99a44c29f\") " Nov 02 20:30:21.936931 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.936906 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d104368e-b2cc-4929-81b0-48a99a44c29f-maas-api-service-ca" pod "d104368e-b2cc-4929-81b0-48a99a44c29f" (UID: "d104368e-b2cc-4929-81b0-48a99a44c29f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:21.937312 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.937262 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d104368e-b2cc-4929-81b0-48a99a44c29f-maas-api-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.938679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:21.938648 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d104368e-b2cc-4929-81b0-48a99a44c29f-kube-api-access-qjzsh" pod "d104368e-b2cc-4929-81b0-48a99a44c29f" (UID: "d104368e-b2cc-4929-81b0-48a99a44c29f"). InnerVolumeSpecName "kube-api-access-qjzsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:22.038886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.038850 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzsh\" (UniqueName: \"kubernetes.io/projected/d104368e-b2cc-4929-81b0-48a99a44c29f-kube-api-access-qjzsh\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:22.069184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.069153 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" Nov 02 20:30:22.069399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.069185 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh" event={"ID":"c3621789-faf8-482e-bf9d-76caea94a3e2","Type":"ContainerDied","Data":"43db0d78e5d7ee65a8d1370fcaa9ee8350b8a15424a2f64ef3a2574b6cbbb521"} Nov 02 20:30:22.069399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.069253 5399 scope.go:122] "RemoveContainer" containerID="553502bf4b1ce71d7cbd9a588b69ebb82748d558c070674eced84e2b75db74ae" Nov 02 20:30:22.072801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.072783 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" Nov 02 20:30:22.072901 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.072811 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9" event={"ID":"d104368e-b2cc-4929-81b0-48a99a44c29f","Type":"ContainerDied","Data":"73da3130e07c27417163fbb468839ff57cc50ab13e947e1b8545217208b7d9c0"} Nov 02 20:30:22.074099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.074079 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" event={"ID":"5eda0988-9984-43df-afa4-5ce94fa58c5a","Type":"ContainerDied","Data":"57afeae69137fb9ce2e4822960f3881ea98bfdfa0f22381cc92bf697d074efdd"} Nov 02 20:30:22.074184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.074097 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8" Nov 02 20:30:22.082931 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.082912 5399 scope.go:122] "RemoveContainer" containerID="86df052e578a210aa35ed30309a7bd447b4d15cb820cefac160e443758f9f833" Nov 02 20:30:22.093914 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.093887 5399 scope.go:122] "RemoveContainer" containerID="50195bd172165269f2369de8c4ae3f68e0bef6ecc56fecd87198815cd2748f89" Nov 02 20:30:22.224578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.224502 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9"] Nov 02 20:30:22.228254 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.228225 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-035e47-29894190-hmzg9"] Nov 02 20:30:22.328449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.328412 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8"] Nov 02 20:30:22.334057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.334029 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-602be312-29894190-rjxl8"] Nov 02 20:30:22.435241 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.435189 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh"] Nov 02 20:30:22.438877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:22.438852 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a51b3f-29894190-kqpwh"] Nov 02 20:30:23.455344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.455308 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj"] Nov 02 20:30:23.456251 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456235 5399 cpu_manager.go:414] "containerMap: removing container" podUID="82d7a94c-d41d-4122-8274-aec68f197f9f" containerName="payload-pre-processing" Nov 02 20:30:23.456345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456255 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d7a94c-d41d-4122-8274-aec68f197f9f" containerName="payload-pre-processing" Nov 02 20:30:23.456345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456289 5399 cpu_manager.go:414] "containerMap: removing container" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerName="cleanup" Nov 02 20:30:23.456345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456298 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerName="cleanup" Nov 02 20:30:23.456345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456310 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" Nov 02 20:30:23.456345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456317 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" Nov 02 20:30:23.456345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456330 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerName="cleanup" Nov 02 20:30:23.456345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456339 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456357 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f8035ece-b691-4025-aab3-182741493aa0" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456364 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8035ece-b691-4025-aab3-182741493aa0" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456375 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f8035ece-b691-4025-aab3-182741493aa0" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456383 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8035ece-b691-4025-aab3-182741493aa0" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456397 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f8035ece-b691-4025-aab3-182741493aa0" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456404 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8035ece-b691-4025-aab3-182741493aa0" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456423 5399 cpu_manager.go:414] "containerMap: removing container" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456430 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456443 5399 cpu_manager.go:414] "containerMap: removing container" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456450 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456465 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456472 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456485 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456492 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456504 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456512 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456530 5399 cpu_manager.go:414] "containerMap: removing container" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456537 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456547 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456554 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456573 5399 cpu_manager.go:414] "containerMap: removing container" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerName="cleanup" Nov 02 20:30:23.456585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456579 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456604 5399 cpu_manager.go:414] "containerMap: removing container" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456612 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456621 5399 cpu_manager.go:414] "containerMap: removing container" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456629 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456639 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456646 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456657 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456663 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456675 5399 cpu_manager.go:414] "containerMap: removing container" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456682 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456694 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456701 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456713 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456720 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456843 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456855 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456865 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456897 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456909 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456918 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456927 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456936 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456946 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456953 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456965 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456974 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456982 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456990 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.456999 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457009 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457019 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457027 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457036 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457044 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457051 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457062 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d7a94c-d41d-4122-8274-aec68f197f9f" containerName="payload-pre-processing" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457073 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457082 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457093 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8035ece-b691-4025-aab3-182741493aa0" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457103 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8035ece-b691-4025-aab3-182741493aa0" containerName="cleanup" Nov 02 20:30:23.457198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.457111 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8035ece-b691-4025-aab3-182741493aa0" containerName="cleanup" Nov 02 20:30:23.462636 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.462616 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.545615 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.545451 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" path="/var/lib/kubelet/pods/5eda0988-9984-43df-afa4-5ce94fa58c5a/volumes" Nov 02 20:30:23.547373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.546353 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" path="/var/lib/kubelet/pods/c3621789-faf8-482e-bf9d-76caea94a3e2/volumes" Nov 02 20:30:23.547373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.547015 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" path="/var/lib/kubelet/pods/d104368e-b2cc-4929-81b0-48a99a44c29f/volumes" Nov 02 20:30:23.551597 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.551563 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj"] Nov 02 20:30:23.558298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.557727 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bb78ba-openshift-default-dockercfg-gbwxt\"" Nov 02 20:30:23.561692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.558823 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-podinfo\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.561692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.558899 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx5tk\" (UniqueName: \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-kube-api-access-zx5tk\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.561692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.558969 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-socket\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.561692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.559034 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-envoy\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.561692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.559076 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istiod-ca-cert\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.561692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.559130 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-data\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.561692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.559166 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-token\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.561692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.559298 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-credential-socket\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.561692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.559352 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-certs\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.661264 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.661211 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-credential-socket\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.661264 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.661309 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-certs\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.661785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.661410 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-podinfo\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.661785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.661480 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx5tk\" (UniqueName: \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-kube-api-access-zx5tk\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.661785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.661561 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-socket\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.661785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.661614 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-envoy\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.661785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.661659 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istiod-ca-cert\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.661785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.661718 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-data\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.661785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.661758 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-token\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.663006 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.662968 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-credential-socket\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.663263 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.663243 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-certs\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.664800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.664773 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istiod-ca-cert\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.665409 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.665373 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-socket\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.665667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.665647 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-data\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.673165 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.673132 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-podinfo\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.674249 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.674227 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-envoy\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.718470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.718395 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-token\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.719706 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.719650 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx5tk\" (UniqueName: \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-kube-api-access-zx5tk\") pod \"e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:23.777717 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:23.777680 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:24.198833 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:30:24.198802 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961e4fe5_5f3a_4d7f_981c_98fe9b42da4f.slice/crio-f1fbfb11da2be5ef0ff22ee127e92bc4a51921023b9192b44a8b39613d20a17b WatchSource:0}: Error finding container f1fbfb11da2be5ef0ff22ee127e92bc4a51921023b9192b44a8b39613d20a17b: Status 404 returned error can't find the container with id f1fbfb11da2be5ef0ff22ee127e92bc4a51921023b9192b44a8b39613d20a17b Nov 02 20:30:24.200480 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:24.200450 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj"] Nov 02 20:30:24.201688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:24.201636 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:30:24.201770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:24.201715 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:30:24.201770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:24.201744 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:30:25.102190 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:25.102155 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" event={"ID":"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f","Type":"ContainerStarted","Data":"a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca"} Nov 02 20:30:25.102689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:25.102211 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" event={"ID":"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f","Type":"ContainerStarted","Data":"f1fbfb11da2be5ef0ff22ee127e92bc4a51921023b9192b44a8b39613d20a17b"} Nov 02 20:30:25.259487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:25.259425 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" podStartSLOduration=2.259409412 podStartE2EDuration="2.259409412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:25.256502746 +0000 UTC m=+2267.867154838" watchObservedRunningTime="2026-11-02 20:30:25.259409412 +0000 UTC m=+2267.870061502" Nov 02 20:30:25.778847 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:25.778814 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:25.785043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:25.785017 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:26.119938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:26.119907 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:26.120935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:26.120911 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:30:34.660458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:34.660419 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7"] Nov 02 20:30:34.706659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:34.706627 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-602be312-nqsj7"] Nov 02 20:30:35.510569 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:35.510534 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468645d2-ef7c-401a-97bd-e7eeb894c964" path="/var/lib/kubelet/pods/468645d2-ef7c-401a-97bd-e7eeb894c964/volumes" Nov 02 20:30:38.534827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:38.534790 5399 scope.go:122] "RemoveContainer" containerID="5efa2c892ed751ae3a6612fff58adc525c229ce6bb84869affdf248a290384a0" Nov 02 20:30:39.831328 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:39.831293 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj"] Nov 02 20:30:39.831778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:39.831541 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" containerID="cri-o://e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298" gracePeriod=30 Nov 02 20:30:40.225034 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:40.224940 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp"] Nov 02 20:30:40.225318 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:40.225217 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" containerID="cri-o://362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9" gracePeriod=30 Nov 02 20:30:42.613259 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:42.613217 5399 patch_prober.go:29] interesting pod/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:30:42.613772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:42.613333 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:30:42.856397 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:42.856365 5399 patch_prober.go:29] interesting pod/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:30:42.856555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:42.856418 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:30:45.124079 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.124052 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:30:45.214333 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214271 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-envoy\") pod \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " Nov 02 20:30:45.214513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214342 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-certs\") pod \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " Nov 02 20:30:45.214513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214380 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-socket\") pod \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " Nov 02 20:30:45.214513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214421 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-credential-socket\") pod \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " Nov 02 20:30:45.214513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214484 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istiod-ca-cert\") pod \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " Nov 02 20:30:45.214769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214524 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-podinfo\") pod \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " Nov 02 20:30:45.214769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214554 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-token\" (UniqueName: \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-token\") pod \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " Nov 02 20:30:45.214769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214638 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-certs" pod "9b1f59bf-d4de-458d-ba74-1aaeb2744888" (UID: "9b1f59bf-d4de-458d-ba74-1aaeb2744888"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.214769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214750 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-credential-socket" pod "9b1f59bf-d4de-458d-ba74-1aaeb2744888" (UID: "9b1f59bf-d4de-458d-ba74-1aaeb2744888"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.214769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214753 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-data\") pod \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " Nov 02 20:30:45.215022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214763 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-socket" pod "9b1f59bf-d4de-458d-ba74-1aaeb2744888" (UID: "9b1f59bf-d4de-458d-ba74-1aaeb2744888"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.215022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214880 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-kube-api-access-lkv9r\" (UniqueName: \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-kube-api-access-lkv9r\") pod \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\" (UID: \"9b1f59bf-d4de-458d-ba74-1aaeb2744888\") " Nov 02 20:30:45.215022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214959 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istiod-ca-cert" pod "9b1f59bf-d4de-458d-ba74-1aaeb2744888" (UID: "9b1f59bf-d4de-458d-ba74-1aaeb2744888"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:45.215022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.214983 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-data" pod "9b1f59bf-d4de-458d-ba74-1aaeb2744888" (UID: "9b1f59bf-d4de-458d-ba74-1aaeb2744888"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.215667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.215640 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.215806 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.215670 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.215806 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.215686 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.215806 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.215701 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.215806 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.215716 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.216724 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.216688 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-envoy" pod "9b1f59bf-d4de-458d-ba74-1aaeb2744888" (UID: "9b1f59bf-d4de-458d-ba74-1aaeb2744888"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.217009 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.216991 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-token" pod "9b1f59bf-d4de-458d-ba74-1aaeb2744888" (UID: "9b1f59bf-d4de-458d-ba74-1aaeb2744888"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:45.217089 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.217037 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-podinfo" pod "9b1f59bf-d4de-458d-ba74-1aaeb2744888" (UID: "9b1f59bf-d4de-458d-ba74-1aaeb2744888"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:45.217129 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.217116 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-kube-api-access-lkv9r" pod "9b1f59bf-d4de-458d-ba74-1aaeb2744888" (UID: "9b1f59bf-d4de-458d-ba74-1aaeb2744888"). InnerVolumeSpecName "kube-api-access-lkv9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:45.271537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.271500 5399 generic.go:356] "Generic (PLEG): container finished" podID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerID="e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298" exitCode=0 Nov 02 20:30:45.271720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.271580 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" Nov 02 20:30:45.271720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.271578 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" event={"ID":"9b1f59bf-d4de-458d-ba74-1aaeb2744888","Type":"ContainerDied","Data":"e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298"} Nov 02 20:30:45.271720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.271662 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj" event={"ID":"9b1f59bf-d4de-458d-ba74-1aaeb2744888","Type":"ContainerDied","Data":"b40bed3a5b8d745ad3a2e35e17079c73115d5b8cd279c597ad3a1c402c95494d"} Nov 02 20:30:45.271720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.271694 5399 scope.go:122] "RemoveContainer" containerID="e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298" Nov 02 20:30:45.318054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.318018 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkv9r\" (UniqueName: \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-kube-api-access-lkv9r\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.318054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.318056 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.318237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.318070 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.318237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.318083 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b1f59bf-d4de-458d-ba74-1aaeb2744888-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.340344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.340305 5399 scope.go:122] "RemoveContainer" containerID="e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298" Nov 02 20:30:45.340781 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:45.340752 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298\": container with ID starting with e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298 not found: ID does not exist" containerID="e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298" Nov 02 20:30:45.340913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.340787 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298"} err="failed to get container status \"e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298\": rpc error: code = NotFound desc = could not find container \"e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298\": container with ID starting with e078c3abdd8b9dbb3be829fdaf8e8d1f500304d7052c2905f970bbd2be4e7298 not found: ID does not exist" Nov 02 20:30:45.438719 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.435399 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj"] Nov 02 20:30:45.440468 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.440437 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-zrrzj"] Nov 02 20:30:45.488372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.488343 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:30:45.508313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.508257 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" path="/var/lib/kubelet/pods/9b1f59bf-d4de-458d-ba74-1aaeb2744888/volumes" Nov 02 20:30:45.623519 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.623486 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-credential-socket\") pod \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " Nov 02 20:30:45.623778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.623612 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-data\") pod \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " Nov 02 20:30:45.623778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.623680 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-certs\") pod \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " Nov 02 20:30:45.623778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.623721 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-podinfo\") pod \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " Nov 02 20:30:45.624962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.623990 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-envoy\") pod \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " Nov 02 20:30:45.624962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.624107 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-token\" (UniqueName: \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-token\") pod \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " Nov 02 20:30:45.624962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.624146 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-kube-api-access-plzfd\" (UniqueName: \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-kube-api-access-plzfd\") pod \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " Nov 02 20:30:45.624962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.624182 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-socket\") pod \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " Nov 02 20:30:45.624962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.624198 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-credential-socket" pod "ef3c6aae-d44a-4a78-adb7-48a19e857afb" (UID: "ef3c6aae-d44a-4a78-adb7-48a19e857afb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.624962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.624217 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istiod-ca-cert\") pod \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\" (UID: \"ef3c6aae-d44a-4a78-adb7-48a19e857afb\") " Nov 02 20:30:45.624962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.624878 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-certs" pod "ef3c6aae-d44a-4a78-adb7-48a19e857afb" (UID: "ef3c6aae-d44a-4a78-adb7-48a19e857afb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.627050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.626050 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.627050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.626074 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.627365 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.627337 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istiod-ca-cert" pod "ef3c6aae-d44a-4a78-adb7-48a19e857afb" (UID: "ef3c6aae-d44a-4a78-adb7-48a19e857afb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:45.627464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.627403 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-socket" pod "ef3c6aae-d44a-4a78-adb7-48a19e857afb" (UID: "ef3c6aae-d44a-4a78-adb7-48a19e857afb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.627561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.627518 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-data" pod "ef3c6aae-d44a-4a78-adb7-48a19e857afb" (UID: "ef3c6aae-d44a-4a78-adb7-48a19e857afb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.627839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.627816 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-podinfo" pod "ef3c6aae-d44a-4a78-adb7-48a19e857afb" (UID: "ef3c6aae-d44a-4a78-adb7-48a19e857afb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:45.628479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.628451 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-token" pod "ef3c6aae-d44a-4a78-adb7-48a19e857afb" (UID: "ef3c6aae-d44a-4a78-adb7-48a19e857afb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:45.628645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.628584 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-kube-api-access-plzfd" pod "ef3c6aae-d44a-4a78-adb7-48a19e857afb" (UID: "ef3c6aae-d44a-4a78-adb7-48a19e857afb"). InnerVolumeSpecName "kube-api-access-plzfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:45.629387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.629358 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-envoy" pod "ef3c6aae-d44a-4a78-adb7-48a19e857afb" (UID: "ef3c6aae-d44a-4a78-adb7-48a19e857afb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:45.727712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.727636 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.727712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.727665 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.727712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.727678 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plzfd\" (UniqueName: \"kubernetes.io/projected/ef3c6aae-d44a-4a78-adb7-48a19e857afb-kube-api-access-plzfd\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.727712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.727688 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.727712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.727696 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.727712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.727704 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:45.727712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:45.727712 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3c6aae-d44a-4a78-adb7-48a19e857afb-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:46.281564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:46.281531 5399 generic.go:356] "Generic (PLEG): container finished" podID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerID="362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9" exitCode=0 Nov 02 20:30:46.282023 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:46.281595 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" Nov 02 20:30:46.282023 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:46.281602 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" event={"ID":"ef3c6aae-d44a-4a78-adb7-48a19e857afb","Type":"ContainerDied","Data":"362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9"} Nov 02 20:30:46.282023 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:46.281658 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp" event={"ID":"ef3c6aae-d44a-4a78-adb7-48a19e857afb","Type":"ContainerDied","Data":"751a16996fc7f8232467dae8dacf4c981d01273d4f5c784b62ff60c41635539d"} Nov 02 20:30:46.282023 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:46.281683 5399 scope.go:122] "RemoveContainer" containerID="362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9" Nov 02 20:30:46.292855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:46.292839 5399 scope.go:122] "RemoveContainer" containerID="362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9" Nov 02 20:30:46.293141 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:46.293109 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9\": container with ID starting with 362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9 not found: ID does not exist" containerID="362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9" Nov 02 20:30:46.293219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:46.293148 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9"} err="failed to get container status \"362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9\": rpc error: code = NotFound desc = could not find container \"362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9\": container with ID starting with 362ec70a5eac72bf21516951cee2a6f6556ebcf99fb54348171f089f488160a9 not found: ID does not exist" Nov 02 20:30:46.436307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:46.436259 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp"] Nov 02 20:30:46.482795 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:46.482763 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-m5qvp"] Nov 02 20:30:47.510967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:47.510931 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" path="/var/lib/kubelet/pods/ef3c6aae-d44a-4a78-adb7-48a19e857afb/volumes" Nov 02 20:30:48.238602 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:30:48.238559 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:30:48.830119 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:30:48.830083 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:31:11.126013 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.125977 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns"] Nov 02 20:31:11.126505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.126181 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" podUID="a132e255-d892-4039-9a76-936c22c94fe1" containerName="maas-api" containerID="cri-o://b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122" gracePeriod=30 Nov 02 20:31:11.449166 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.449138 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:31:11.516250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.516193 5399 generic.go:356] "Generic (PLEG): container finished" podID="a132e255-d892-4039-9a76-936c22c94fe1" containerID="b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122" exitCode=0 Nov 02 20:31:11.516510 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.516496 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" Nov 02 20:31:11.520776 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.520749 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" event={"ID":"a132e255-d892-4039-9a76-936c22c94fe1","Type":"ContainerDied","Data":"b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122"} Nov 02 20:31:11.520918 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.520808 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns" event={"ID":"a132e255-d892-4039-9a76-936c22c94fe1","Type":"ContainerDied","Data":"861ac54e05084b8cb47463169a812939f17bbb752cb4e7165b178658dd685b73"} Nov 02 20:31:11.520918 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.520839 5399 scope.go:122] "RemoveContainer" containerID="b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122" Nov 02 20:31:11.531595 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.531572 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-metrics-certs\") pod \"a132e255-d892-4039-9a76-936c22c94fe1\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " Nov 02 20:31:11.531700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.531632 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-maas-api-tls\") pod \"a132e255-d892-4039-9a76-936c22c94fe1\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " Nov 02 20:31:11.531700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.531669 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a132e255-d892-4039-9a76-936c22c94fe1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a132e255-d892-4039-9a76-936c22c94fe1-openshift-service-ca\") pod \"a132e255-d892-4039-9a76-936c22c94fe1\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " Nov 02 20:31:11.531841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.531776 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a132e255-d892-4039-9a76-936c22c94fe1-kube-api-access-w2cp9\" (UniqueName: \"kubernetes.io/projected/a132e255-d892-4039-9a76-936c22c94fe1-kube-api-access-w2cp9\") pod \"a132e255-d892-4039-9a76-936c22c94fe1\" (UID: \"a132e255-d892-4039-9a76-936c22c94fe1\") " Nov 02 20:31:11.531962 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.531900 5399 scope.go:122] "RemoveContainer" containerID="b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122" Nov 02 20:31:11.532161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.532134 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a132e255-d892-4039-9a76-936c22c94fe1-openshift-service-ca" pod "a132e255-d892-4039-9a76-936c22c94fe1" (UID: "a132e255-d892-4039-9a76-936c22c94fe1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:11.532234 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:11.532195 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122\": container with ID starting with b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122 not found: ID does not exist" containerID="b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122" Nov 02 20:31:11.532291 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.532231 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122"} err="failed to get container status \"b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122\": rpc error: code = NotFound desc = could not find container \"b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122\": container with ID starting with b56dd61bba8dbacd53ab556b889c14787801415a7893f4d6cd8dff47a5dfc122 not found: ID does not exist" Nov 02 20:31:11.532501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.532483 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a132e255-d892-4039-9a76-936c22c94fe1-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.534008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.533962 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-maas-api-tls" pod "a132e255-d892-4039-9a76-936c22c94fe1" (UID: "a132e255-d892-4039-9a76-936c22c94fe1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:11.534008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.533988 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-metrics-certs" pod "a132e255-d892-4039-9a76-936c22c94fe1" (UID: "a132e255-d892-4039-9a76-936c22c94fe1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:11.534226 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.534210 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a132e255-d892-4039-9a76-936c22c94fe1-kube-api-access-w2cp9" pod "a132e255-d892-4039-9a76-936c22c94fe1" (UID: "a132e255-d892-4039-9a76-936c22c94fe1"). InnerVolumeSpecName "kube-api-access-w2cp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:11.633421 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.633388 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.633421 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.633416 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a132e255-d892-4039-9a76-936c22c94fe1-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.633421 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.633427 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2cp9\" (UniqueName: \"kubernetes.io/projected/a132e255-d892-4039-9a76-936c22c94fe1-kube-api-access-w2cp9\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:11.931954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.931916 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns"] Nov 02 20:31:11.935172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:11.935147 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c76017-677b8795c-p25ns"] Nov 02 20:31:12.224916 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.224832 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48"] Nov 02 20:31:12.225348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.225059 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" podUID="99461eb1-62b6-4369-8aec-3cb288042a0b" containerName="maas-api" containerID="cri-o://478b1cf24bebf8b12c9480cd4441ba777dfd6d0e7f1c4de4d9cc063504ba94e7" gracePeriod=30 Nov 02 20:31:12.525583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.525553 5399 generic.go:356] "Generic (PLEG): container finished" podID="99461eb1-62b6-4369-8aec-3cb288042a0b" containerID="478b1cf24bebf8b12c9480cd4441ba777dfd6d0e7f1c4de4d9cc063504ba94e7" exitCode=0 Nov 02 20:31:12.525734 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.525624 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" event={"ID":"99461eb1-62b6-4369-8aec-3cb288042a0b","Type":"ContainerDied","Data":"478b1cf24bebf8b12c9480cd4441ba777dfd6d0e7f1c4de4d9cc063504ba94e7"} Nov 02 20:31:12.525734 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.525675 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" event={"ID":"99461eb1-62b6-4369-8aec-3cb288042a0b","Type":"ContainerDied","Data":"a03f6a8cea933f1e0ce322c7d3a207192cc9888732dab9b410879ed6a0d9955d"} Nov 02 20:31:12.525734 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.525695 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a03f6a8cea933f1e0ce322c7d3a207192cc9888732dab9b410879ed6a0d9955d" Nov 02 20:31:12.540186 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.540164 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:31:12.650378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.650340 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-metrics-certs\") pod \"99461eb1-62b6-4369-8aec-3cb288042a0b\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " Nov 02 20:31:12.650591 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.650441 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99461eb1-62b6-4369-8aec-3cb288042a0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99461eb1-62b6-4369-8aec-3cb288042a0b-openshift-service-ca\") pod \"99461eb1-62b6-4369-8aec-3cb288042a0b\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " Nov 02 20:31:12.650591 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.650469 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-maas-api-tls\") pod \"99461eb1-62b6-4369-8aec-3cb288042a0b\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " Nov 02 20:31:12.650924 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.650803 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99461eb1-62b6-4369-8aec-3cb288042a0b-kube-api-access-vwg7r\" (UniqueName: \"kubernetes.io/projected/99461eb1-62b6-4369-8aec-3cb288042a0b-kube-api-access-vwg7r\") pod \"99461eb1-62b6-4369-8aec-3cb288042a0b\" (UID: \"99461eb1-62b6-4369-8aec-3cb288042a0b\") " Nov 02 20:31:12.650924 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.650853 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99461eb1-62b6-4369-8aec-3cb288042a0b-openshift-service-ca" pod "99461eb1-62b6-4369-8aec-3cb288042a0b" (UID: "99461eb1-62b6-4369-8aec-3cb288042a0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:12.651787 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.651721 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99461eb1-62b6-4369-8aec-3cb288042a0b-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:12.653211 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.653182 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-maas-api-tls" pod "99461eb1-62b6-4369-8aec-3cb288042a0b" (UID: "99461eb1-62b6-4369-8aec-3cb288042a0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:12.654408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.654382 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-metrics-certs" pod "99461eb1-62b6-4369-8aec-3cb288042a0b" (UID: "99461eb1-62b6-4369-8aec-3cb288042a0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:12.655451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.655424 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99461eb1-62b6-4369-8aec-3cb288042a0b-kube-api-access-vwg7r" pod "99461eb1-62b6-4369-8aec-3cb288042a0b" (UID: "99461eb1-62b6-4369-8aec-3cb288042a0b"). InnerVolumeSpecName "kube-api-access-vwg7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:12.753582 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.753544 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:12.753582 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.753576 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwg7r\" (UniqueName: \"kubernetes.io/projected/99461eb1-62b6-4369-8aec-3cb288042a0b-kube-api-access-vwg7r\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:12.753582 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:12.753587 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99461eb1-62b6-4369-8aec-3cb288042a0b-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:13.509522 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:13.509492 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a132e255-d892-4039-9a76-936c22c94fe1" path="/var/lib/kubelet/pods/a132e255-d892-4039-9a76-936c22c94fe1/volumes" Nov 02 20:31:13.536506 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:13.536478 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48" Nov 02 20:31:13.728062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:13.728029 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48"] Nov 02 20:31:13.774475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:13.774401 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-429b45-5fc689cb49-s2z48"] Nov 02 20:31:14.476008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:14.475973 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv"] Nov 02 20:31:14.522044 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:14.522011 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-429b45-nlcnv"] Nov 02 20:31:15.511744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:15.511708 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8acfc830-3a5f-43ff-a28f-de8032f61754" path="/var/lib/kubelet/pods/8acfc830-3a5f-43ff-a28f-de8032f61754/volumes" Nov 02 20:31:15.512218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:15.512204 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99461eb1-62b6-4369-8aec-3cb288042a0b" path="/var/lib/kubelet/pods/99461eb1-62b6-4369-8aec-3cb288042a0b/volumes" Nov 02 20:31:18.239394 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:18.239354 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:31:19.853388 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:19.853351 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs"] Nov 02 20:31:19.853878 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:19.853675 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerName="istio-proxy" containerID="cri-o://03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910" gracePeriod=30 Nov 02 20:31:20.709792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.709756 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc"] Nov 02 20:31:20.710421 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710405 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" Nov 02 20:31:20.710494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710422 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" Nov 02 20:31:20.710494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710432 5399 cpu_manager.go:414] "containerMap: removing container" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerName="cleanup" Nov 02 20:31:20.710494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710437 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eda0988-9984-43df-afa4-5ce94fa58c5a" containerName="cleanup" Nov 02 20:31:20.710494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710452 5399 cpu_manager.go:414] "containerMap: removing container" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" Nov 02 20:31:20.710494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710458 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" Nov 02 20:31:20.710494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710471 5399 cpu_manager.go:414] "containerMap: removing container" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" containerName="cleanup" Nov 02 20:31:20.710494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710477 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e81e4f5-0154-4327-9de3-6690783b632a" containerName="cleanup" Nov 02 20:31:20.710494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710484 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerName="cleanup" Nov 02 20:31:20.710494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710488 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710499 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710504 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3621789-faf8-482e-bf9d-76caea94a3e2" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710513 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710517 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db637dd-249a-4b6f-9a3c-5b00cd9bc7f5" containerName="istio-proxy" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710529 5399 cpu_manager.go:414] "containerMap: removing container" podUID="a132e255-d892-4039-9a76-936c22c94fe1" containerName="maas-api" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710533 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a132e255-d892-4039-9a76-936c22c94fe1" containerName="maas-api" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710540 5399 cpu_manager.go:414] "containerMap: removing container" podUID="99461eb1-62b6-4369-8aec-3cb288042a0b" containerName="maas-api" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710544 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99461eb1-62b6-4369-8aec-3cb288042a0b" containerName="maas-api" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710553 5399 cpu_manager.go:414] "containerMap: removing container" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710559 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0d8e94-5394-46a9-aceb-08532e885e00" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710567 5399 cpu_manager.go:414] "containerMap: removing container" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710572 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710590 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710594 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc569dc-8a4b-4774-9b77-a020e1d821a2" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710675 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3c6aae-d44a-4a78-adb7-48a19e857afb" containerName="istio-proxy" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710686 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1f59bf-d4de-458d-ba74-1aaeb2744888" containerName="istio-proxy" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710693 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d104368e-b2cc-4929-81b0-48a99a44c29f" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710701 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a132e255-d892-4039-9a76-936c22c94fe1" containerName="maas-api" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710707 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="99461eb1-62b6-4369-8aec-3cb288042a0b" containerName="maas-api" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710713 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bbab00-c103-433f-a3ec-90d6b1faec7b" containerName="cleanup" Nov 02 20:31:20.710764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.710721 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e254e9b3-e008-4c18-b6b1-786188414fa9" containerName="cleanup" Nov 02 20:31:20.715584 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.715563 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:20.772711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.772678 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc"] Nov 02 20:31:20.805471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.805434 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:20.805648 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.805592 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:20.805813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.805787 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrzn\" (UniqueName: \"kubernetes.io/projected/319d44b8-1f13-41d9-ba98-7222d9575fe3-kube-api-access-lwrzn\") pod \"maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:20.813825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.813800 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:20.813951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.813903 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:31:20.907773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.907736 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:20.908248 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.907827 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:20.908248 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.907924 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrzn\" (UniqueName: \"kubernetes.io/projected/319d44b8-1f13-41d9-ba98-7222d9575fe3-kube-api-access-lwrzn\") pod \"maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:20.908675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.908651 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:20.910210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.910190 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:20.960855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:20.960782 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrzn\" (UniqueName: \"kubernetes.io/projected/319d44b8-1f13-41d9-ba98-7222d9575fe3-kube-api-access-lwrzn\") pod \"maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:21.029938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:21.029905 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:21.401198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:21.401166 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc"] Nov 02 20:31:21.405232 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:31:21.405204 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319d44b8_1f13_41d9_ba98_7222d9575fe3.slice/crio-b8f787440e63545048cbc486619989b81fd438b8af54a31cf7b50401527d2386 WatchSource:0}: Error finding container b8f787440e63545048cbc486619989b81fd438b8af54a31cf7b50401527d2386: Status 404 returned error can't find the container with id b8f787440e63545048cbc486619989b81fd438b8af54a31cf7b50401527d2386 Nov 02 20:31:21.642094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:21.642050 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" event={"ID":"319d44b8-1f13-41d9-ba98-7222d9575fe3","Type":"ContainerStarted","Data":"b8f787440e63545048cbc486619989b81fd438b8af54a31cf7b50401527d2386"} Nov 02 20:31:22.333711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.333676 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f"] Nov 02 20:31:22.338040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.338014 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.390599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.390567 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f"] Nov 02 20:31:22.439502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.439465 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.439680 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.439577 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprnh\" (UniqueName: \"kubernetes.io/projected/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-kube-api-access-xprnh\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.439680 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.439660 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.541378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.541345 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprnh\" (UniqueName: \"kubernetes.io/projected/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-kube-api-access-xprnh\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.541545 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.541399 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.541601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.541561 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.542205 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.542181 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.543822 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.543798 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.596638 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.596568 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprnh\" (UniqueName: \"kubernetes.io/projected/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-kube-api-access-xprnh\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.651367 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.651329 5399 generic.go:356] "Generic (PLEG): container finished" podID="319d44b8-1f13-41d9-ba98-7222d9575fe3" containerID="01eeab4b220ff859c534e0c89559d60b845772769d2d2babda54234a8f4e3747" exitCode=0 Nov 02 20:31:22.651554 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.651391 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" event={"ID":"319d44b8-1f13-41d9-ba98-7222d9575fe3","Type":"ContainerDied","Data":"01eeab4b220ff859c534e0c89559d60b845772769d2d2babda54234a8f4e3747"} Nov 02 20:31:22.652455 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.652432 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:22.841014 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.840975 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp"] Nov 02 20:31:22.841247 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.841215 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerName="istio-proxy" containerID="cri-o://605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a" gracePeriod=30 Nov 02 20:31:22.859185 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.859118 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd"] Nov 02 20:31:22.859382 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:22.859358 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" podUID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" containerName="istio-proxy" containerID="cri-o://925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403" gracePeriod=30 Nov 02 20:31:23.031860 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.029651 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f"] Nov 02 20:31:23.052579 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:31:23.052548 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa4bac8_8fec_4905_beb6_33c77e9f1aef.slice/crio-abff80e44096fd6bfe9b4de37d806cc4114d1428b151b5b93afff77679533fb9 WatchSource:0}: Error finding container abff80e44096fd6bfe9b4de37d806cc4114d1428b151b5b93afff77679533fb9: Status 404 returned error can't find the container with id abff80e44096fd6bfe9b4de37d806cc4114d1428b151b5b93afff77679533fb9 Nov 02 20:31:23.660560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.660522 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" event={"ID":"9aa4bac8-8fec-4905-beb6-33c77e9f1aef","Type":"ContainerStarted","Data":"a54913674b2e6d540e541deebb5319e9044e9bd05c362aedb3ed0c45dcac8f72"} Nov 02 20:31:23.660958 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.660582 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" event={"ID":"9aa4bac8-8fec-4905-beb6-33c77e9f1aef","Type":"ContainerStarted","Data":"abff80e44096fd6bfe9b4de37d806cc4114d1428b151b5b93afff77679533fb9"} Nov 02 20:31:23.769252 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.769192 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" podStartSLOduration=1.769174129 podStartE2EDuration="1.769174129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:23.768638189 +0000 UTC m=+2326.379290281" watchObservedRunningTime="2026-11-02 20:31:23.769174129 +0000 UTC m=+2326.379826220" Nov 02 20:31:23.817986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.817960 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:23.961559 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.961525 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-openshift-ca\") pod \"319d44b8-1f13-41d9-ba98-7222d9575fe3\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " Nov 02 20:31:23.961750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.961637 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319d44b8-1f13-41d9-ba98-7222d9575fe3-kube-api-access-lwrzn\" (UniqueName: \"kubernetes.io/projected/319d44b8-1f13-41d9-ba98-7222d9575fe3-kube-api-access-lwrzn\") pod \"319d44b8-1f13-41d9-ba98-7222d9575fe3\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " Nov 02 20:31:23.961871 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.961852 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-xks-ca\") pod \"319d44b8-1f13-41d9-ba98-7222d9575fe3\" (UID: \"319d44b8-1f13-41d9-ba98-7222d9575fe3\") " Nov 02 20:31:23.962023 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.961991 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-openshift-ca" pod "319d44b8-1f13-41d9-ba98-7222d9575fe3" (UID: "319d44b8-1f13-41d9-ba98-7222d9575fe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:23.962552 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.962527 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:23.964026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.963999 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-xks-ca" pod "319d44b8-1f13-41d9-ba98-7222d9575fe3" (UID: "319d44b8-1f13-41d9-ba98-7222d9575fe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:23.964146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:23.964040 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319d44b8-1f13-41d9-ba98-7222d9575fe3-kube-api-access-lwrzn" pod "319d44b8-1f13-41d9-ba98-7222d9575fe3" (UID: "319d44b8-1f13-41d9-ba98-7222d9575fe3"). InnerVolumeSpecName "kube-api-access-lwrzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:24.064353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:24.064315 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319d44b8-1f13-41d9-ba98-7222d9575fe3-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:24.064353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:24.064347 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwrzn\" (UniqueName: \"kubernetes.io/projected/319d44b8-1f13-41d9-ba98-7222d9575fe3-kube-api-access-lwrzn\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:24.669521 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:24.669482 5399 generic.go:356] "Generic (PLEG): container finished" podID="9aa4bac8-8fec-4905-beb6-33c77e9f1aef" containerID="a54913674b2e6d540e541deebb5319e9044e9bd05c362aedb3ed0c45dcac8f72" exitCode=0 Nov 02 20:31:24.669969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:24.669560 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" event={"ID":"9aa4bac8-8fec-4905-beb6-33c77e9f1aef","Type":"ContainerDied","Data":"a54913674b2e6d540e541deebb5319e9044e9bd05c362aedb3ed0c45dcac8f72"} Nov 02 20:31:24.671059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:24.671035 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" Nov 02 20:31:24.671195 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:24.671057 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc" event={"ID":"319d44b8-1f13-41d9-ba98-7222d9575fe3","Type":"ContainerDied","Data":"b8f787440e63545048cbc486619989b81fd438b8af54a31cf7b50401527d2386"} Nov 02 20:31:24.671195 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:24.671096 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f787440e63545048cbc486619989b81fd438b8af54a31cf7b50401527d2386" Nov 02 20:31:25.124038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.124013 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:31:25.281073 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.280983 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-data\") pod \"df2e6659-eb29-4b1e-b1e9-17b25342836d\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " Nov 02 20:31:25.281073 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281057 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-certs\") pod \"df2e6659-eb29-4b1e-b1e9-17b25342836d\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " Nov 02 20:31:25.281364 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281096 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2e6659-eb29-4b1e-b1e9-17b25342836d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2e6659-eb29-4b1e-b1e9-17b25342836d-istiod-ca-cert\") pod \"df2e6659-eb29-4b1e-b1e9-17b25342836d\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " Nov 02 20:31:25.281364 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281151 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-envoy\") pod \"df2e6659-eb29-4b1e-b1e9-17b25342836d\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " Nov 02 20:31:25.281489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281373 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-credential-socket\") pod \"df2e6659-eb29-4b1e-b1e9-17b25342836d\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " Nov 02 20:31:25.281489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281395 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-certs" pod "df2e6659-eb29-4b1e-b1e9-17b25342836d" (UID: "df2e6659-eb29-4b1e-b1e9-17b25342836d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:25.281600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281463 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-kube-api-access-zr4s9\" (UniqueName: \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-kube-api-access-zr4s9\") pod \"df2e6659-eb29-4b1e-b1e9-17b25342836d\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " Nov 02 20:31:25.281600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281531 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-podinfo\") pod \"df2e6659-eb29-4b1e-b1e9-17b25342836d\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " Nov 02 20:31:25.281713 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281605 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-token\" (UniqueName: \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-token\") pod \"df2e6659-eb29-4b1e-b1e9-17b25342836d\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " Nov 02 20:31:25.281713 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281643 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2e6659-eb29-4b1e-b1e9-17b25342836d-istiod-ca-cert" pod "df2e6659-eb29-4b1e-b1e9-17b25342836d" (UID: "df2e6659-eb29-4b1e-b1e9-17b25342836d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:25.281713 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281646 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-socket\") pod \"df2e6659-eb29-4b1e-b1e9-17b25342836d\" (UID: \"df2e6659-eb29-4b1e-b1e9-17b25342836d\") " Nov 02 20:31:25.281875 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281845 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-socket" pod "df2e6659-eb29-4b1e-b1e9-17b25342836d" (UID: "df2e6659-eb29-4b1e-b1e9-17b25342836d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:25.282246 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.281929 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-credential-socket" pod "df2e6659-eb29-4b1e-b1e9-17b25342836d" (UID: "df2e6659-eb29-4b1e-b1e9-17b25342836d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:25.282246 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.282004 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-data" pod "df2e6659-eb29-4b1e-b1e9-17b25342836d" (UID: "df2e6659-eb29-4b1e-b1e9-17b25342836d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:25.282583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.282557 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.282666 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.282589 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.282666 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.282604 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.282666 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.282618 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.282666 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.282632 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2e6659-eb29-4b1e-b1e9-17b25342836d-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.284065 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.284040 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-envoy" pod "df2e6659-eb29-4b1e-b1e9-17b25342836d" (UID: "df2e6659-eb29-4b1e-b1e9-17b25342836d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:25.284468 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.284446 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-kube-api-access-zr4s9" pod "df2e6659-eb29-4b1e-b1e9-17b25342836d" (UID: "df2e6659-eb29-4b1e-b1e9-17b25342836d"). InnerVolumeSpecName "kube-api-access-zr4s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:25.284939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.284905 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-podinfo" pod "df2e6659-eb29-4b1e-b1e9-17b25342836d" (UID: "df2e6659-eb29-4b1e-b1e9-17b25342836d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:25.285061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.285040 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-token" pod "df2e6659-eb29-4b1e-b1e9-17b25342836d" (UID: "df2e6659-eb29-4b1e-b1e9-17b25342836d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:25.384516 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.384478 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr4s9\" (UniqueName: \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-kube-api-access-zr4s9\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.384516 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.384509 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.384516 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.384520 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.384760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.384533 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2e6659-eb29-4b1e-b1e9-17b25342836d-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.680062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.680027 5399 generic.go:356] "Generic (PLEG): container finished" podID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerID="03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910" exitCode=0 Nov 02 20:31:25.680602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.680076 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" Nov 02 20:31:25.680602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.680117 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" event={"ID":"df2e6659-eb29-4b1e-b1e9-17b25342836d","Type":"ContainerDied","Data":"03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910"} Nov 02 20:31:25.680602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.680181 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs" event={"ID":"df2e6659-eb29-4b1e-b1e9-17b25342836d","Type":"ContainerDied","Data":"987cf68aefb28731db92afafbb800977aefb2edf0ccfebf7671fb55354767ba4"} Nov 02 20:31:25.680602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.680286 5399 scope.go:122] "RemoveContainer" containerID="03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910" Nov 02 20:31:25.693323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.693269 5399 scope.go:122] "RemoveContainer" containerID="03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910" Nov 02 20:31:25.693615 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:25.693595 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910\": container with ID starting with 03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910 not found: ID does not exist" containerID="03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910" Nov 02 20:31:25.693695 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.693629 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910"} err="failed to get container status \"03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910\": rpc error: code = NotFound desc = could not find container \"03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910\": container with ID starting with 03e7542af7aed476b7d049a14fa1f7ac16681b01774804b018de83a3dd8f3910 not found: ID does not exist" Nov 02 20:31:25.836047 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.836017 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs"] Nov 02 20:31:25.848151 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.848126 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:25.882026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.881994 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c76017-openshift-default-5f5d56d997-9lqjs"] Nov 02 20:31:25.993454 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.993360 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-xks-ca\") pod \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " Nov 02 20:31:25.993638 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.993505 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-openshift-ca\") pod \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " Nov 02 20:31:25.993638 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.993571 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-kube-api-access-xprnh\" (UniqueName: \"kubernetes.io/projected/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-kube-api-access-xprnh\") pod \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\" (UID: \"9aa4bac8-8fec-4905-beb6-33c77e9f1aef\") " Nov 02 20:31:25.993902 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.993868 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-openshift-ca" pod "9aa4bac8-8fec-4905-beb6-33c77e9f1aef" (UID: "9aa4bac8-8fec-4905-beb6-33c77e9f1aef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:25.994298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.994241 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:25.996033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.996002 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-xks-ca" pod "9aa4bac8-8fec-4905-beb6-33c77e9f1aef" (UID: "9aa4bac8-8fec-4905-beb6-33c77e9f1aef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:25.998478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:25.998447 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-kube-api-access-xprnh" pod "9aa4bac8-8fec-4905-beb6-33c77e9f1aef" (UID: "9aa4bac8-8fec-4905-beb6-33c77e9f1aef"). InnerVolumeSpecName "kube-api-access-xprnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:26.096083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:26.096050 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xprnh\" (UniqueName: \"kubernetes.io/projected/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-kube-api-access-xprnh\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:26.096083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:26.096082 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aa4bac8-8fec-4905-beb6-33c77e9f1aef-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:26.688368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:26.688326 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" event={"ID":"9aa4bac8-8fec-4905-beb6-33c77e9f1aef","Type":"ContainerDied","Data":"abff80e44096fd6bfe9b4de37d806cc4114d1428b151b5b93afff77679533fb9"} Nov 02 20:31:26.688368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:26.688355 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f" Nov 02 20:31:26.688897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:26.688387 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abff80e44096fd6bfe9b4de37d806cc4114d1428b151b5b93afff77679533fb9" Nov 02 20:31:27.509709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:27.509674 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" path="/var/lib/kubelet/pods/df2e6659-eb29-4b1e-b1e9-17b25342836d/volumes" Nov 02 20:31:28.136438 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.136404 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:31:28.186572 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.186546 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:31:28.226198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226167 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-data\") pod \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " Nov 02 20:31:28.226396 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226243 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-kube-api-access-tmdcv\" (UniqueName: \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-kube-api-access-tmdcv\") pod \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " Nov 02 20:31:28.226396 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226356 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-credential-socket\") pod \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " Nov 02 20:31:28.226396 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226395 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-certs\") pod \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " Nov 02 20:31:28.226589 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226471 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istiod-ca-cert\") pod \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " Nov 02 20:31:28.226589 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226537 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-data" pod "3daeb5a8-fef8-4c6d-a94c-0116902df49b" (UID: "3daeb5a8-fef8-4c6d-a94c-0116902df49b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.226589 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226557 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-envoy\") pod \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " Nov 02 20:31:28.226732 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226633 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-credential-socket" pod "3daeb5a8-fef8-4c6d-a94c-0116902df49b" (UID: "3daeb5a8-fef8-4c6d-a94c-0116902df49b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.226732 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226717 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-certs" pod "3daeb5a8-fef8-4c6d-a94c-0116902df49b" (UID: "3daeb5a8-fef8-4c6d-a94c-0116902df49b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.226835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226795 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-podinfo\") pod \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " Nov 02 20:31:28.226893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226847 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-socket\") pod \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " Nov 02 20:31:28.226946 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226911 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istiod-ca-cert" pod "3daeb5a8-fef8-4c6d-a94c-0116902df49b" (UID: "3daeb5a8-fef8-4c6d-a94c-0116902df49b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:28.226997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.226941 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-token\" (UniqueName: \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-token\") pod \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\" (UID: \"3daeb5a8-fef8-4c6d-a94c-0116902df49b\") " Nov 02 20:31:28.227190 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.227166 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-socket" pod "3daeb5a8-fef8-4c6d-a94c-0116902df49b" (UID: "3daeb5a8-fef8-4c6d-a94c-0116902df49b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.227722 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.227696 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.227722 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.227727 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.227898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.227742 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.227898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.227757 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.227898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.227771 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.228993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.228967 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-kube-api-access-tmdcv" pod "3daeb5a8-fef8-4c6d-a94c-0116902df49b" (UID: "3daeb5a8-fef8-4c6d-a94c-0116902df49b"). InnerVolumeSpecName "kube-api-access-tmdcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.229205 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.229179 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-podinfo" pod "3daeb5a8-fef8-4c6d-a94c-0116902df49b" (UID: "3daeb5a8-fef8-4c6d-a94c-0116902df49b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:28.229306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.229212 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-token" pod "3daeb5a8-fef8-4c6d-a94c-0116902df49b" (UID: "3daeb5a8-fef8-4c6d-a94c-0116902df49b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.229306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.229300 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-envoy" pod "3daeb5a8-fef8-4c6d-a94c-0116902df49b" (UID: "3daeb5a8-fef8-4c6d-a94c-0116902df49b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.328686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.328656 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-envoy\") pod \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " Nov 02 20:31:28.328897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.328710 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-kube-api-access-hvlr7\" (UniqueName: \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-kube-api-access-hvlr7\") pod \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " Nov 02 20:31:28.328897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.328747 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-certs\") pod \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " Nov 02 20:31:28.329032 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.328929 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-token\" (UniqueName: \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-token\") pod \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " Nov 02 20:31:28.329032 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.329018 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-podinfo\") pod \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " Nov 02 20:31:28.329145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.329053 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istiod-ca-cert\") pod \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " Nov 02 20:31:28.329145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.329105 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-data\") pod \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " Nov 02 20:31:28.329145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.329137 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-socket\") pod \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " Nov 02 20:31:28.329807 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.329153 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-certs" pod "dc859550-a1bd-4c32-96ae-1c22edd82f6d" (UID: "dc859550-a1bd-4c32-96ae-1c22edd82f6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.329807 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.329266 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-credential-socket\") pod \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\" (UID: \"dc859550-a1bd-4c32-96ae-1c22edd82f6d\") " Nov 02 20:31:28.329807 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.329502 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-data" pod "dc859550-a1bd-4c32-96ae-1c22edd82f6d" (UID: "dc859550-a1bd-4c32-96ae-1c22edd82f6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.329807 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.329531 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istiod-ca-cert" pod "dc859550-a1bd-4c32-96ae-1c22edd82f6d" (UID: "dc859550-a1bd-4c32-96ae-1c22edd82f6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:28.329807 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.329738 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-socket" pod "dc859550-a1bd-4c32-96ae-1c22edd82f6d" (UID: "dc859550-a1bd-4c32-96ae-1c22edd82f6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.329807 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.329763 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-credential-socket" pod "dc859550-a1bd-4c32-96ae-1c22edd82f6d" (UID: "dc859550-a1bd-4c32-96ae-1c22edd82f6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.330237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.330108 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.330237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.330133 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmdcv\" (UniqueName: \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-kube-api-access-tmdcv\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.330237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.330147 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.330237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.330163 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.330237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.330176 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.330237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.330192 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.330237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.330206 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.330237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.330220 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.330237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.330234 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daeb5a8-fef8-4c6d-a94c-0116902df49b-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.331332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.331299 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-kube-api-access-hvlr7" pod "dc859550-a1bd-4c32-96ae-1c22edd82f6d" (UID: "dc859550-a1bd-4c32-96ae-1c22edd82f6d"). InnerVolumeSpecName "kube-api-access-hvlr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.331605 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.331582 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-podinfo" pod "dc859550-a1bd-4c32-96ae-1c22edd82f6d" (UID: "dc859550-a1bd-4c32-96ae-1c22edd82f6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:28.331692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.331664 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-token" pod "dc859550-a1bd-4c32-96ae-1c22edd82f6d" (UID: "dc859550-a1bd-4c32-96ae-1c22edd82f6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.331784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.331769 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-envoy" pod "dc859550-a1bd-4c32-96ae-1c22edd82f6d" (UID: "dc859550-a1bd-4c32-96ae-1c22edd82f6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:28.432145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.432059 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.432145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.432094 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvlr7\" (UniqueName: \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-kube-api-access-hvlr7\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.432145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.432111 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.432145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.432124 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc859550-a1bd-4c32-96ae-1c22edd82f6d-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.706933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.706848 5399 generic.go:356] "Generic (PLEG): container finished" podID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" containerID="925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403" exitCode=0 Nov 02 20:31:28.706933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.706901 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" Nov 02 20:31:28.707157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.706931 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" event={"ID":"dc859550-a1bd-4c32-96ae-1c22edd82f6d","Type":"ContainerDied","Data":"925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403"} Nov 02 20:31:28.707157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.706990 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd" event={"ID":"dc859550-a1bd-4c32-96ae-1c22edd82f6d","Type":"ContainerDied","Data":"33ac5a2b4a166019dee9b404717b431c767a69c69576f0bcfc147a87a096b95a"} Nov 02 20:31:28.707157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.707019 5399 scope.go:122] "RemoveContainer" containerID="925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403" Nov 02 20:31:28.708687 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.708662 5399 generic.go:356] "Generic (PLEG): container finished" podID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerID="605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a" exitCode=0 Nov 02 20:31:28.708791 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.708714 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" Nov 02 20:31:28.708791 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.708745 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" event={"ID":"3daeb5a8-fef8-4c6d-a94c-0116902df49b","Type":"ContainerDied","Data":"605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a"} Nov 02 20:31:28.708881 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.708794 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp" event={"ID":"3daeb5a8-fef8-4c6d-a94c-0116902df49b","Type":"ContainerDied","Data":"d4508d387b0e225ec9c5e2d2645d899414d2a745f5e7118b0b06afeecd85bf5e"} Nov 02 20:31:28.719725 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.719705 5399 scope.go:122] "RemoveContainer" containerID="925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403" Nov 02 20:31:28.720012 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:28.719991 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403\": container with ID starting with 925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403 not found: ID does not exist" containerID="925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403" Nov 02 20:31:28.720098 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.720024 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403"} err="failed to get container status \"925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403\": rpc error: code = NotFound desc = could not find container \"925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403\": container with ID starting with 925c664a4ea3e8e022c6bd218af7f4951e0c46f7175290dcd09878c8f73a1403 not found: ID does not exist" Nov 02 20:31:28.720098 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.720048 5399 scope.go:122] "RemoveContainer" containerID="605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a" Nov 02 20:31:28.730949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.730925 5399 scope.go:122] "RemoveContainer" containerID="605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a" Nov 02 20:31:28.731308 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:28.731283 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a\": container with ID starting with 605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a not found: ID does not exist" containerID="605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a" Nov 02 20:31:28.731373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.731314 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a"} err="failed to get container status \"605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a\": rpc error: code = NotFound desc = could not find container \"605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a\": container with ID starting with 605ec1cfe3b1da1241bc1133021cfd7d9778e3ef3930bd260056692e71c8dc1a not found: ID does not exist" Nov 02 20:31:28.861598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.861564 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp"] Nov 02 20:31:28.868646 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.868613 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-m6cgp"] Nov 02 20:31:28.969302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.969203 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd"] Nov 02 20:31:28.972678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:28.972656 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-602be312-openshift-default-5d565d78d9-bv9wd"] Nov 02 20:31:29.509904 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.509874 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" path="/var/lib/kubelet/pods/3daeb5a8-fef8-4c6d-a94c-0116902df49b/volumes" Nov 02 20:31:29.510453 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.510437 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" path="/var/lib/kubelet/pods/dc859550-a1bd-4c32-96ae-1c22edd82f6d/volumes" Nov 02 20:31:29.987869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.987833 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p"] Nov 02 20:31:29.988465 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988450 5399 cpu_manager.go:414] "containerMap: removing container" podUID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" containerName="istio-proxy" Nov 02 20:31:29.988541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988466 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" containerName="istio-proxy" Nov 02 20:31:29.988541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988481 5399 cpu_manager.go:414] "containerMap: removing container" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerName="istio-proxy" Nov 02 20:31:29.988541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988486 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerName="istio-proxy" Nov 02 20:31:29.988541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988501 5399 cpu_manager.go:414] "containerMap: removing container" podUID="319d44b8-1f13-41d9-ba98-7222d9575fe3" containerName="revoke-keys" Nov 02 20:31:29.988541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988507 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319d44b8-1f13-41d9-ba98-7222d9575fe3" containerName="revoke-keys" Nov 02 20:31:29.988541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988523 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerName="istio-proxy" Nov 02 20:31:29.988541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988528 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerName="istio-proxy" Nov 02 20:31:29.988541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988542 5399 cpu_manager.go:414] "containerMap: removing container" podUID="9aa4bac8-8fec-4905-beb6-33c77e9f1aef" containerName="revoke-keys" Nov 02 20:31:29.988782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988548 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa4bac8-8fec-4905-beb6-33c77e9f1aef" containerName="revoke-keys" Nov 02 20:31:29.988782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988630 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="319d44b8-1f13-41d9-ba98-7222d9575fe3" containerName="revoke-keys" Nov 02 20:31:29.988782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988640 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3daeb5a8-fef8-4c6d-a94c-0116902df49b" containerName="istio-proxy" Nov 02 20:31:29.988782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988648 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa4bac8-8fec-4905-beb6-33c77e9f1aef" containerName="revoke-keys" Nov 02 20:31:29.988782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988657 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc859550-a1bd-4c32-96ae-1c22edd82f6d" containerName="istio-proxy" Nov 02 20:31:29.988782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.988664 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2e6659-eb29-4b1e-b1e9-17b25342836d" containerName="istio-proxy" Nov 02 20:31:29.993659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:29.993639 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.045193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.045162 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p"] Nov 02 20:31:30.059433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.059398 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78p62\" (UniqueName: \"kubernetes.io/projected/492a1b11-831e-424b-8620-0e9476094a71-kube-api-access-78p62\") pod \"maas-api-revoke-sub-08ad024e-dk57p\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.059603 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.059448 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492a1b11-831e-424b-8620-0e9476094a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ad024e-dk57p\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.059696 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.059679 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492a1b11-831e-424b-8620-0e9476094a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ad024e-dk57p\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.086539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.086501 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:30.086705 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.086557 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:31:30.161180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.161149 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492a1b11-831e-424b-8620-0e9476094a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ad024e-dk57p\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.161366 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.161264 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78p62\" (UniqueName: \"kubernetes.io/projected/492a1b11-831e-424b-8620-0e9476094a71-kube-api-access-78p62\") pod \"maas-api-revoke-sub-08ad024e-dk57p\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.161440 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.161366 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492a1b11-831e-424b-8620-0e9476094a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ad024e-dk57p\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.162018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.161985 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492a1b11-831e-424b-8620-0e9476094a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ad024e-dk57p\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.163629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.163608 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492a1b11-831e-424b-8620-0e9476094a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ad024e-dk57p\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.214691 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.214654 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78p62\" (UniqueName: \"kubernetes.io/projected/492a1b11-831e-424b-8620-0e9476094a71-kube-api-access-78p62\") pod \"maas-api-revoke-sub-08ad024e-dk57p\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.307236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.307213 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:30.674999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.674966 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p"] Nov 02 20:31:30.677033 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:31:30.677009 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492a1b11_831e_424b_8620_0e9476094a71.slice/crio-44f84049b6ca52538837a94b23e97cbec345a0dd33d5c74d8b0bdc7568f78546 WatchSource:0}: Error finding container 44f84049b6ca52538837a94b23e97cbec345a0dd33d5c74d8b0bdc7568f78546: Status 404 returned error can't find the container with id 44f84049b6ca52538837a94b23e97cbec345a0dd33d5c74d8b0bdc7568f78546 Nov 02 20:31:30.726777 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:30.726748 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" event={"ID":"492a1b11-831e-424b-8620-0e9476094a71","Type":"ContainerStarted","Data":"44f84049b6ca52538837a94b23e97cbec345a0dd33d5c74d8b0bdc7568f78546"} Nov 02 20:31:31.735463 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:31.735425 5399 generic.go:356] "Generic (PLEG): container finished" podID="492a1b11-831e-424b-8620-0e9476094a71" containerID="c0c01b847ff34c0d667f3ec3f13c6318cab0559e0bc5f32884da831826cbf686" exitCode=0 Nov 02 20:31:31.735866 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:31.735505 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" event={"ID":"492a1b11-831e-424b-8620-0e9476094a71","Type":"ContainerDied","Data":"c0c01b847ff34c0d667f3ec3f13c6318cab0559e0bc5f32884da831826cbf686"} Nov 02 20:31:32.922036 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:32.922007 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:32.999073 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:32.998987 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492a1b11-831e-424b-8620-0e9476094a71-kube-api-access-78p62\" (UniqueName: \"kubernetes.io/projected/492a1b11-831e-424b-8620-0e9476094a71-kube-api-access-78p62\") pod \"492a1b11-831e-424b-8620-0e9476094a71\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " Nov 02 20:31:32.999229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:32.999143 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492a1b11-831e-424b-8620-0e9476094a71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492a1b11-831e-424b-8620-0e9476094a71-maas-api-openshift-ca\") pod \"492a1b11-831e-424b-8620-0e9476094a71\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " Nov 02 20:31:32.999229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:32.999193 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492a1b11-831e-424b-8620-0e9476094a71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492a1b11-831e-424b-8620-0e9476094a71-maas-api-xks-ca\") pod \"492a1b11-831e-424b-8620-0e9476094a71\" (UID: \"492a1b11-831e-424b-8620-0e9476094a71\") " Nov 02 20:31:32.999618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:32.999594 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492a1b11-831e-424b-8620-0e9476094a71-maas-api-openshift-ca" pod "492a1b11-831e-424b-8620-0e9476094a71" (UID: "492a1b11-831e-424b-8620-0e9476094a71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:32.999751 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:32.999731 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492a1b11-831e-424b-8620-0e9476094a71-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:33.001235 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:33.001213 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492a1b11-831e-424b-8620-0e9476094a71-maas-api-xks-ca" pod "492a1b11-831e-424b-8620-0e9476094a71" (UID: "492a1b11-831e-424b-8620-0e9476094a71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:33.001336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:33.001264 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492a1b11-831e-424b-8620-0e9476094a71-kube-api-access-78p62" pod "492a1b11-831e-424b-8620-0e9476094a71" (UID: "492a1b11-831e-424b-8620-0e9476094a71"). InnerVolumeSpecName "kube-api-access-78p62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:33.100674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:33.100632 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492a1b11-831e-424b-8620-0e9476094a71-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:33.100674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:33.100668 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78p62\" (UniqueName: \"kubernetes.io/projected/492a1b11-831e-424b-8620-0e9476094a71-kube-api-access-78p62\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:33.822997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:33.822957 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" Nov 02 20:31:33.823238 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:33.823019 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p" event={"ID":"492a1b11-831e-424b-8620-0e9476094a71","Type":"ContainerDied","Data":"44f84049b6ca52538837a94b23e97cbec345a0dd33d5c74d8b0bdc7568f78546"} Nov 02 20:31:33.823238 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:33.823112 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44f84049b6ca52538837a94b23e97cbec345a0dd33d5c74d8b0bdc7568f78546" Nov 02 20:31:34.850346 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:34.850307 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww"] Nov 02 20:31:36.137924 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:36.137884 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:31:38.856410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.856378 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd"] Nov 02 20:31:38.856993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.856979 5399 cpu_manager.go:414] "containerMap: removing container" podUID="492a1b11-831e-424b-8620-0e9476094a71" containerName="revoke-keys" Nov 02 20:31:38.857059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.856994 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492a1b11-831e-424b-8620-0e9476094a71" containerName="revoke-keys" Nov 02 20:31:38.857135 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.857126 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="492a1b11-831e-424b-8620-0e9476094a71" containerName="revoke-keys" Nov 02 20:31:38.862696 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.862673 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:38.907828 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.907792 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd"] Nov 02 20:31:38.915313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.915206 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee30a99-5158-4da4-a8a6-95c9fd155810-openshift-service-ca\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:38.915499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.915469 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-metrics-certs\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:38.915712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.915686 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgdd9\" (UniqueName: \"kubernetes.io/projected/eee30a99-5158-4da4-a8a6-95c9fd155810-kube-api-access-rgdd9\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:38.915844 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.915817 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-maas-api-tls\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:38.951301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:38.951239 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-65f28f-serving-cert\"" Nov 02 20:31:39.017888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.017847 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-maas-api-tls\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:39.018447 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.018416 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee30a99-5158-4da4-a8a6-95c9fd155810-openshift-service-ca\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:39.019325 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:39.018190 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-serving-cert: secret "maas-api-e2e-inf-a-65f28f-serving-cert" not found Nov 02 20:31:39.019442 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:39.019409 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-maas-api-tls podName:eee30a99-5158-4da4-a8a6-95c9fd155810 nodeName:}" failed. No retries permitted until 2026-11-02 20:31:39.519382961 +0000 UTC m=+2342.130035043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-maas-api-tls") pod "maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" (UID: "eee30a99-5158-4da4-a8a6-95c9fd155810") : secret "maas-api-e2e-inf-a-65f28f-serving-cert" not found Nov 02 20:31:39.019865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.019595 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-metrics-certs\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:39.019865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.019659 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgdd9\" (UniqueName: \"kubernetes.io/projected/eee30a99-5158-4da4-a8a6-95c9fd155810-kube-api-access-rgdd9\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:39.019865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.019797 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee30a99-5158-4da4-a8a6-95c9fd155810-openshift-service-ca\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:39.022090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.022068 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-metrics-certs\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:39.076162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.076124 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgdd9\" (UniqueName: \"kubernetes.io/projected/eee30a99-5158-4da4-a8a6-95c9fd155810-kube-api-access-rgdd9\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:39.547299 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.547228 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-maas-api-tls\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:39.551892 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.551859 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-maas-api-tls\") pod \"maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:39.777204 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.777167 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:39.854364 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:39.852807 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" podUID="3cc9167c-b871-405c-b8e9-fc901deb83b7" containerName="payload-pre-processing" containerID="cri-o://77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269" gracePeriod=25 Nov 02 20:31:40.175184 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:31:40.174564 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee30a99_5158_4da4_a8a6_95c9fd155810.slice/crio-418472c442100cc088b2cb40a8e1e86b1d445b0f0aaba9d178e2d07ab7386445 WatchSource:0}: Error finding container 418472c442100cc088b2cb40a8e1e86b1d445b0f0aaba9d178e2d07ab7386445: Status 404 returned error can't find the container with id 418472c442100cc088b2cb40a8e1e86b1d445b0f0aaba9d178e2d07ab7386445 Nov 02 20:31:40.175699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.175313 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd"] Nov 02 20:31:40.176195 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.176172 5399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:31:40.192139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.192115 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:31:40.362923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.362885 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc9167c-b871-405c-b8e9-fc901deb83b7-kube-api-access-h66pt\" (UniqueName: \"kubernetes.io/projected/3cc9167c-b871-405c-b8e9-fc901deb83b7-kube-api-access-h66pt\") pod \"3cc9167c-b871-405c-b8e9-fc901deb83b7\" (UID: \"3cc9167c-b871-405c-b8e9-fc901deb83b7\") " Nov 02 20:31:40.363223 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.363201 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cc9167c-b871-405c-b8e9-fc901deb83b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc9167c-b871-405c-b8e9-fc901deb83b7-plugins-config-volume\") pod \"3cc9167c-b871-405c-b8e9-fc901deb83b7\" (UID: \"3cc9167c-b871-405c-b8e9-fc901deb83b7\") " Nov 02 20:31:40.363671 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.363639 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc9167c-b871-405c-b8e9-fc901deb83b7-plugins-config-volume" pod "3cc9167c-b871-405c-b8e9-fc901deb83b7" (UID: "3cc9167c-b871-405c-b8e9-fc901deb83b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:40.363825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.363807 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc9167c-b871-405c-b8e9-fc901deb83b7-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:40.365960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.365923 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc9167c-b871-405c-b8e9-fc901deb83b7-kube-api-access-h66pt" pod "3cc9167c-b871-405c-b8e9-fc901deb83b7" (UID: "3cc9167c-b871-405c-b8e9-fc901deb83b7"). InnerVolumeSpecName "kube-api-access-h66pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:40.465627 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.465594 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h66pt\" (UniqueName: \"kubernetes.io/projected/3cc9167c-b871-405c-b8e9-fc901deb83b7-kube-api-access-h66pt\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:40.886553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.886517 5399 generic.go:356] "Generic (PLEG): container finished" podID="3cc9167c-b871-405c-b8e9-fc901deb83b7" containerID="77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269" exitCode=0 Nov 02 20:31:40.886772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.886569 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" Nov 02 20:31:40.886772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.886609 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" event={"ID":"3cc9167c-b871-405c-b8e9-fc901deb83b7","Type":"ContainerDied","Data":"77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269"} Nov 02 20:31:40.886772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.886666 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww" event={"ID":"3cc9167c-b871-405c-b8e9-fc901deb83b7","Type":"ContainerDied","Data":"280cfbb662bf0a32a13b1db8f405e06348090bfe40c28e928dfd1ce1b9809a6f"} Nov 02 20:31:40.886772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.886695 5399 scope.go:122] "RemoveContainer" containerID="77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269" Nov 02 20:31:40.888199 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.888161 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" event={"ID":"eee30a99-5158-4da4-a8a6-95c9fd155810","Type":"ContainerStarted","Data":"418472c442100cc088b2cb40a8e1e86b1d445b0f0aaba9d178e2d07ab7386445"} Nov 02 20:31:40.897956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.897933 5399 scope.go:122] "RemoveContainer" containerID="77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269" Nov 02 20:31:40.898301 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:40.898228 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269\": container with ID starting with 77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269 not found: ID does not exist" containerID="77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269" Nov 02 20:31:40.898371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:40.898318 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269"} err="failed to get container status \"77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269\": rpc error: code = NotFound desc = could not find container \"77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269\": container with ID starting with 77d4b265550ee54c15deb09a5b4fb0d39ccec74e31573f173df045b12465d269 not found: ID does not exist" Nov 02 20:31:41.074527 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.074328 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww"] Nov 02 20:31:41.078076 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.078056 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-be1348-79c7d594c-8xjww"] Nov 02 20:31:41.256330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.256291 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g"] Nov 02 20:31:41.256935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.256918 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3cc9167c-b871-405c-b8e9-fc901deb83b7" containerName="payload-pre-processing" Nov 02 20:31:41.257023 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.256937 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc9167c-b871-405c-b8e9-fc901deb83b7" containerName="payload-pre-processing" Nov 02 20:31:41.257079 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.257025 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc9167c-b871-405c-b8e9-fc901deb83b7" containerName="payload-pre-processing" Nov 02 20:31:41.260580 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.260563 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.303048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.301431 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g"] Nov 02 20:31:41.309753 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.309718 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:41.309932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.309758 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:31:41.380078 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.380045 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2q5q\" (UniqueName: \"kubernetes.io/projected/ff4d7297-e99a-4906-abd0-af7566f01302-kube-api-access-q2q5q\") pod \"maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.380250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.380095 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.380250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.380189 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.482072 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.481986 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2q5q\" (UniqueName: \"kubernetes.io/projected/ff4d7297-e99a-4906-abd0-af7566f01302-kube-api-access-q2q5q\") pod \"maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.482072 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.482047 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.482298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.482090 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.482912 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.482889 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.484554 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.484531 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.508793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.508766 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc9167c-b871-405c-b8e9-fc901deb83b7" path="/var/lib/kubelet/pods/3cc9167c-b871-405c-b8e9-fc901deb83b7/volumes" Nov 02 20:31:41.553933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.553903 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2q5q\" (UniqueName: \"kubernetes.io/projected/ff4d7297-e99a-4906-abd0-af7566f01302-kube-api-access-q2q5q\") pod \"maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.575909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.575879 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:41.897170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.897132 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" event={"ID":"eee30a99-5158-4da4-a8a6-95c9fd155810","Type":"ContainerStarted","Data":"7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6"} Nov 02 20:31:41.897404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.897218 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:41.975082 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:41.975032 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g"] Nov 02 20:31:42.009362 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:42.009321 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" podStartSLOduration=3.169364942 podStartE2EDuration="4.009307907s" totalImagesPullingTime="839.942965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:42.00909252 +0000 UTC m=+2344.619744623" watchObservedRunningTime="2026-11-02 20:31:42.009307907 +0000 UTC m=+2344.619959992" Nov 02 20:31:42.905708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:42.905658 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" event={"ID":"ff4d7297-e99a-4906-abd0-af7566f01302","Type":"ContainerStarted","Data":"4ef517afe6792a6200d45fb057fb217afa9a64e6dd528491db4ca46dda9c5cd4"} Nov 02 20:31:42.906138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:42.905728 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" event={"ID":"ff4d7297-e99a-4906-abd0-af7566f01302","Type":"ContainerStarted","Data":"58d630a15b1e91d88dbd981458475780a8275de5ea450c53ed4079e066d60819"} Nov 02 20:31:43.012115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:43.012059 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" podStartSLOduration=2.012045699 podStartE2EDuration="2.012045699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:43.010799009 +0000 UTC m=+2345.621451100" watchObservedRunningTime="2026-11-02 20:31:43.012045699 +0000 UTC m=+2345.622697792" Nov 02 20:31:43.943302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:43.938604 5399 generic.go:356] "Generic (PLEG): container finished" podID="ff4d7297-e99a-4906-abd0-af7566f01302" containerID="4ef517afe6792a6200d45fb057fb217afa9a64e6dd528491db4ca46dda9c5cd4" exitCode=0 Nov 02 20:31:43.943302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:43.938740 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" event={"ID":"ff4d7297-e99a-4906-abd0-af7566f01302","Type":"ContainerDied","Data":"4ef517afe6792a6200d45fb057fb217afa9a64e6dd528491db4ca46dda9c5cd4"} Nov 02 20:31:45.101814 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.101790 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:45.144259 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.144207 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4d7297-e99a-4906-abd0-af7566f01302-kube-api-access-q2q5q\" (UniqueName: \"kubernetes.io/projected/ff4d7297-e99a-4906-abd0-af7566f01302-kube-api-access-q2q5q\") pod \"ff4d7297-e99a-4906-abd0-af7566f01302\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " Nov 02 20:31:45.144453 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.144430 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-xks-ca\") pod \"ff4d7297-e99a-4906-abd0-af7566f01302\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " Nov 02 20:31:45.144629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.144607 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-openshift-ca\") pod \"ff4d7297-e99a-4906-abd0-af7566f01302\" (UID: \"ff4d7297-e99a-4906-abd0-af7566f01302\") " Nov 02 20:31:45.144969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.144943 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-openshift-ca" pod "ff4d7297-e99a-4906-abd0-af7566f01302" (UID: "ff4d7297-e99a-4906-abd0-af7566f01302"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:45.145398 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.145377 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:45.146663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.146633 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-xks-ca" pod "ff4d7297-e99a-4906-abd0-af7566f01302" (UID: "ff4d7297-e99a-4906-abd0-af7566f01302"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:45.146810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.146765 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4d7297-e99a-4906-abd0-af7566f01302-kube-api-access-q2q5q" pod "ff4d7297-e99a-4906-abd0-af7566f01302" (UID: "ff4d7297-e99a-4906-abd0-af7566f01302"). InnerVolumeSpecName "kube-api-access-q2q5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:45.247390 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.247303 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4d7297-e99a-4906-abd0-af7566f01302-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:45.247390 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.247346 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2q5q\" (UniqueName: \"kubernetes.io/projected/ff4d7297-e99a-4906-abd0-af7566f01302-kube-api-access-q2q5q\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:45.601238 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.601204 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f"] Nov 02 20:31:45.604390 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.604369 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-be1348-lt99f"] Nov 02 20:31:45.711049 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.711014 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68"] Nov 02 20:31:45.956573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.956541 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" Nov 02 20:31:45.956749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.956565 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g" event={"ID":"ff4d7297-e99a-4906-abd0-af7566f01302","Type":"ContainerDied","Data":"58d630a15b1e91d88dbd981458475780a8275de5ea450c53ed4079e066d60819"} Nov 02 20:31:45.956749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:45.956626 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58d630a15b1e91d88dbd981458475780a8275de5ea450c53ed4079e066d60819" Nov 02 20:31:47.510202 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:47.510096 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aa4bac8-8fec-4905-beb6-33c77e9f1aef" path="/var/lib/kubelet/pods/9aa4bac8-8fec-4905-beb6-33c77e9f1aef/volumes" Nov 02 20:31:47.911484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:47.911452 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:31:47.973310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:47.973264 5399 generic.go:356] "Generic (PLEG): container finished" podID="32778100-8a2b-46c9-b507-2273c72e10af" containerID="699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57" exitCode=0 Nov 02 20:31:47.973515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:47.973339 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" event={"ID":"32778100-8a2b-46c9-b507-2273c72e10af","Type":"ContainerDied","Data":"699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57"} Nov 02 20:31:47.973801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:47.973786 5399 scope.go:122] "RemoveContainer" containerID="699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57" Nov 02 20:31:47.975570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:47.975543 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:31:47.975641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:47.975612 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:31:47.975678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:47.975650 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:31:48.241545 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:48.238718 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:31:48.983106 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:48.983068 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" event={"ID":"32778100-8a2b-46c9-b507-2273c72e10af","Type":"ContainerStarted","Data":"335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5"} Nov 02 20:31:49.262359 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:49.261687 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:31:49.262359 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:49.261827 5399 patch_prober.go:29] interesting pod/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:31:49.262359 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:49.261871 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 20:31:50.257973 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:50.257938 5399 patch_prober.go:29] interesting pod/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:31:50.258449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:50.258015 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 20:31:51.257370 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:51.257335 5399 patch_prober.go:29] interesting pod/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:31:51.257554 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:51.257389 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 20:31:52.259332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:52.258704 5399 patch_prober.go:29] interesting pod/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:31:52.259332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:52.258789 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 20:31:52.266617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:52.266586 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx"] Nov 02 20:31:52.267107 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:52.266789 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" containerID="cri-o://9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca" gracePeriod=30 Nov 02 20:31:52.856075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:52.856036 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm"] Nov 02 20:31:52.971590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:52.971550 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq"] Nov 02 20:31:53.101032 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.100991 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd"] Nov 02 20:31:53.101728 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.101704 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ff4d7297-e99a-4906-abd0-af7566f01302" containerName="revoke-keys" Nov 02 20:31:53.101728 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.101728 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4d7297-e99a-4906-abd0-af7566f01302" containerName="revoke-keys" Nov 02 20:31:53.101902 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.101870 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4d7297-e99a-4906-abd0-af7566f01302" containerName="revoke-keys" Nov 02 20:31:53.105475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.105445 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.155861 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.155569 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:53.156248 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.156224 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:31:53.159655 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.159632 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd"] Nov 02 20:31:53.201374 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.201330 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.201636 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.201615 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48h6\" (UniqueName: \"kubernetes.io/projected/36a6ac3b-bf3c-4dd5-8844-2397e570117c-kube-api-access-h48h6\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.201885 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.201868 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.258653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.258621 5399 patch_prober.go:29] interesting pod/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:31:53.258827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.258675 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 20:31:53.304294 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.304242 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.304889 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.304842 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48h6\" (UniqueName: \"kubernetes.io/projected/36a6ac3b-bf3c-4dd5-8844-2397e570117c-kube-api-access-h48h6\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.305215 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.305196 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.305564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.305528 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.315425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.315397 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.365920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.365836 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48h6\" (UniqueName: \"kubernetes.io/projected/36a6ac3b-bf3c-4dd5-8844-2397e570117c-kube-api-access-h48h6\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.419449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.419410 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:53.875601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:53.875566 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd"] Nov 02 20:31:53.876400 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:31:53.876373 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a6ac3b_bf3c_4dd5_8844_2397e570117c.slice/crio-cc14f34dae1c7bb34c37f72126f85543b6092f8a68f992e83a79eaa5ac26a479 WatchSource:0}: Error finding container cc14f34dae1c7bb34c37f72126f85543b6092f8a68f992e83a79eaa5ac26a479: Status 404 returned error can't find the container with id cc14f34dae1c7bb34c37f72126f85543b6092f8a68f992e83a79eaa5ac26a479 Nov 02 20:31:54.052473 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:54.046653 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" event={"ID":"36a6ac3b-bf3c-4dd5-8844-2397e570117c","Type":"ContainerStarted","Data":"cc14f34dae1c7bb34c37f72126f85543b6092f8a68f992e83a79eaa5ac26a479"} Nov 02 20:31:54.257869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:54.257786 5399 patch_prober.go:29] interesting pod/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 20:31:54.257869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:54.257839 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 20:31:55.055619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:55.055582 5399 generic.go:356] "Generic (PLEG): container finished" podID="36a6ac3b-bf3c-4dd5-8844-2397e570117c" containerID="c35c3b8ff720b3373eb42b46fac843dcf55737595f53629b4bdb7e640e614016" exitCode=0 Nov 02 20:31:55.056055 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:55.055642 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" event={"ID":"36a6ac3b-bf3c-4dd5-8844-2397e570117c","Type":"ContainerDied","Data":"c35c3b8ff720b3373eb42b46fac843dcf55737595f53629b4bdb7e640e614016"} Nov 02 20:31:55.262905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:55.262826 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:31:55.263122 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:55.263104 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:31:55.263814 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:55.263794 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:31:56.214775 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:56.214750 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:56.364345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:56.364249 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-xks-ca\") pod \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " Nov 02 20:31:56.364531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:56.364356 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-openshift-ca\") pod \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " Nov 02 20:31:56.364531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:56.364393 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a6ac3b-bf3c-4dd5-8844-2397e570117c-kube-api-access-h48h6\" (UniqueName: \"kubernetes.io/projected/36a6ac3b-bf3c-4dd5-8844-2397e570117c-kube-api-access-h48h6\") pod \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\" (UID: \"36a6ac3b-bf3c-4dd5-8844-2397e570117c\") " Nov 02 20:31:56.364773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:56.364742 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-openshift-ca" pod "36a6ac3b-bf3c-4dd5-8844-2397e570117c" (UID: "36a6ac3b-bf3c-4dd5-8844-2397e570117c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:56.365162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:56.365139 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:56.366572 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:56.366542 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-xks-ca" pod "36a6ac3b-bf3c-4dd5-8844-2397e570117c" (UID: "36a6ac3b-bf3c-4dd5-8844-2397e570117c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:56.366679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:56.366599 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a6ac3b-bf3c-4dd5-8844-2397e570117c-kube-api-access-h48h6" pod "36a6ac3b-bf3c-4dd5-8844-2397e570117c" (UID: "36a6ac3b-bf3c-4dd5-8844-2397e570117c"). InnerVolumeSpecName "kube-api-access-h48h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:56.466402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:56.466365 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a6ac3b-bf3c-4dd5-8844-2397e570117c-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:56.466402 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:56.466399 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h48h6\" (UniqueName: \"kubernetes.io/projected/36a6ac3b-bf3c-4dd5-8844-2397e570117c-kube-api-access-h48h6\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.073540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.073503 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" event={"ID":"36a6ac3b-bf3c-4dd5-8844-2397e570117c","Type":"ContainerDied","Data":"cc14f34dae1c7bb34c37f72126f85543b6092f8a68f992e83a79eaa5ac26a479"} Nov 02 20:31:57.073540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.073535 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd" Nov 02 20:31:57.073762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.073560 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc14f34dae1c7bb34c37f72126f85543b6092f8a68f992e83a79eaa5ac26a479" Nov 02 20:31:57.544475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.544449 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:31:57.682038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.681951 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-credential-socket\") pod \"5fd261a1-a473-4c79-a8d8-71d19923c944\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " Nov 02 20:31:57.682038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682002 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-envoy\") pod \"5fd261a1-a473-4c79-a8d8-71d19923c944\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " Nov 02 20:31:57.682324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682134 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd261a1-a473-4c79-a8d8-71d19923c944-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd261a1-a473-4c79-a8d8-71d19923c944-istiod-ca-cert\") pod \"5fd261a1-a473-4c79-a8d8-71d19923c944\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " Nov 02 20:31:57.682324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682180 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-token\" (UniqueName: \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-token\") pod \"5fd261a1-a473-4c79-a8d8-71d19923c944\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " Nov 02 20:31:57.682324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682203 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-credential-socket" pod "5fd261a1-a473-4c79-a8d8-71d19923c944" (UID: "5fd261a1-a473-4c79-a8d8-71d19923c944"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:57.682324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682214 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-kube-api-access-vgsnx\" (UniqueName: \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-kube-api-access-vgsnx\") pod \"5fd261a1-a473-4c79-a8d8-71d19923c944\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " Nov 02 20:31:57.682578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682361 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-socket\") pod \"5fd261a1-a473-4c79-a8d8-71d19923c944\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " Nov 02 20:31:57.682578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682522 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd261a1-a473-4c79-a8d8-71d19923c944-istiod-ca-cert" pod "5fd261a1-a473-4c79-a8d8-71d19923c944" (UID: "5fd261a1-a473-4c79-a8d8-71d19923c944"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:57.682578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682534 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-data\") pod \"5fd261a1-a473-4c79-a8d8-71d19923c944\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " Nov 02 20:31:57.682578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682578 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-certs\") pod \"5fd261a1-a473-4c79-a8d8-71d19923c944\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " Nov 02 20:31:57.682788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682664 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-podinfo\") pod \"5fd261a1-a473-4c79-a8d8-71d19923c944\" (UID: \"5fd261a1-a473-4c79-a8d8-71d19923c944\") " Nov 02 20:31:57.682788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682729 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-socket" pod "5fd261a1-a473-4c79-a8d8-71d19923c944" (UID: "5fd261a1-a473-4c79-a8d8-71d19923c944"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:57.682945 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.682922 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-data" pod "5fd261a1-a473-4c79-a8d8-71d19923c944" (UID: "5fd261a1-a473-4c79-a8d8-71d19923c944"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:57.683135 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.683111 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-certs" pod "5fd261a1-a473-4c79-a8d8-71d19923c944" (UID: "5fd261a1-a473-4c79-a8d8-71d19923c944"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:57.683482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.683457 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd261a1-a473-4c79-a8d8-71d19923c944-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.683581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.683486 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.683581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.683501 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.683581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.683516 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.683581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.683530 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.685318 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.685263 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-kube-api-access-vgsnx" pod "5fd261a1-a473-4c79-a8d8-71d19923c944" (UID: "5fd261a1-a473-4c79-a8d8-71d19923c944"). InnerVolumeSpecName "kube-api-access-vgsnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:57.685506 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.685474 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-podinfo" pod "5fd261a1-a473-4c79-a8d8-71d19923c944" (UID: "5fd261a1-a473-4c79-a8d8-71d19923c944"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:57.687179 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.687157 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-token" pod "5fd261a1-a473-4c79-a8d8-71d19923c944" (UID: "5fd261a1-a473-4c79-a8d8-71d19923c944"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:57.689315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.689295 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-envoy" pod "5fd261a1-a473-4c79-a8d8-71d19923c944" (UID: "5fd261a1-a473-4c79-a8d8-71d19923c944"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:57.786317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.785481 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.786317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.785521 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.786317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.785537 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.786317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.785552 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgsnx\" (UniqueName: \"kubernetes.io/projected/5fd261a1-a473-4c79-a8d8-71d19923c944-kube-api-access-vgsnx\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.857059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.856986 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" podUID="829f41dc-f27c-4d36-9670-eaebac12f098" containerName="payload-processing" containerID="cri-o://a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd" gracePeriod=25 Nov 02 20:31:57.972634 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:57.972514 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" podUID="e7f94892-433e-4a0e-8a6b-004aadfcab15" containerName="payload-pre-processing" containerID="cri-o://a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef" gracePeriod=25 Nov 02 20:31:58.096243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.094370 5399 generic.go:356] "Generic (PLEG): container finished" podID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerID="9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca" exitCode=0 Nov 02 20:31:58.096243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.094689 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" event={"ID":"5fd261a1-a473-4c79-a8d8-71d19923c944","Type":"ContainerDied","Data":"9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca"} Nov 02 20:31:58.096243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.094789 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" event={"ID":"5fd261a1-a473-4c79-a8d8-71d19923c944","Type":"ContainerDied","Data":"63feb3fff5d50e33111d5ef37684d4084d57e1ce9bbb6f5bc4131fe7bc339f54"} Nov 02 20:31:58.096243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.094823 5399 scope.go:122] "RemoveContainer" containerID="9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca" Nov 02 20:31:58.096243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.094999 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx" Nov 02 20:31:58.119041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.119011 5399 scope.go:122] "RemoveContainer" containerID="a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a" Nov 02 20:31:58.131084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.131053 5399 scope.go:122] "RemoveContainer" containerID="9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca" Nov 02 20:31:58.131359 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:58.131332 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca\": container with ID starting with 9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca not found: ID does not exist" containerID="9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca" Nov 02 20:31:58.131430 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.131363 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca"} err="failed to get container status \"9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca\": rpc error: code = NotFound desc = could not find container \"9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca\": container with ID starting with 9c0d65909a13ae1c8295f2f985587088d58aeb8bada57b8c3d76f7dcf22283ca not found: ID does not exist" Nov 02 20:31:58.131430 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.131383 5399 scope.go:122] "RemoveContainer" containerID="a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a" Nov 02 20:31:58.131687 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:31:58.131662 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a\": container with ID starting with a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a not found: ID does not exist" containerID="a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a" Nov 02 20:31:58.131744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.131696 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a"} err="failed to get container status \"a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a\": rpc error: code = NotFound desc = could not find container \"a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a\": container with ID starting with a4048215f7fac6d27f21e348d27401f87ce47ae5e66c2c4565d6d39e3eda059a not found: ID does not exist" Nov 02 20:31:58.250912 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.250832 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx"] Nov 02 20:31:58.297388 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:58.297351 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-be1348-openshift-default-7dd6b8f6dc-5rtlx"] Nov 02 20:31:59.509744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:31:59.509662 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" path="/var/lib/kubelet/pods/5fd261a1-a473-4c79-a8d8-71d19923c944/volumes" Nov 02 20:32:00.733082 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:00.733041 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" containerID="cri-o://38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8" gracePeriod=45 Nov 02 20:32:01.008798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.008773 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:32:01.027081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.027047 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-tmp-dir\") pod \"fa70b62b-4591-401c-bb78-5dfdde976042\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " Nov 02 20:32:01.027208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.027130 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-dshm\") pod \"fa70b62b-4591-401c-bb78-5dfdde976042\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " Nov 02 20:32:01.027208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.027152 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa70b62b-4591-401c-bb78-5dfdde976042-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa70b62b-4591-401c-bb78-5dfdde976042-tls-certs\") pod \"fa70b62b-4591-401c-bb78-5dfdde976042\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " Nov 02 20:32:01.027208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.027190 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa70b62b-4591-401c-bb78-5dfdde976042-kube-api-access-x64th\" (UniqueName: \"kubernetes.io/projected/fa70b62b-4591-401c-bb78-5dfdde976042-kube-api-access-x64th\") pod \"fa70b62b-4591-401c-bb78-5dfdde976042\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " Nov 02 20:32:01.027416 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.027222 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-model-cache\") pod \"fa70b62b-4591-401c-bb78-5dfdde976042\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " Nov 02 20:32:01.027416 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.027338 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-home\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-home\") pod \"fa70b62b-4591-401c-bb78-5dfdde976042\" (UID: \"fa70b62b-4591-401c-bb78-5dfdde976042\") " Nov 02 20:32:01.027416 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.027388 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-tmp-dir" pod "fa70b62b-4591-401c-bb78-5dfdde976042" (UID: "fa70b62b-4591-401c-bb78-5dfdde976042"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.027668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.027640 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-model-cache" pod "fa70b62b-4591-401c-bb78-5dfdde976042" (UID: "fa70b62b-4591-401c-bb78-5dfdde976042"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.027878 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.027818 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-home" pod "fa70b62b-4591-401c-bb78-5dfdde976042" (UID: "fa70b62b-4591-401c-bb78-5dfdde976042"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.028158 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.028135 5399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-tmp-dir\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.028240 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.028164 5399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-model-cache\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.028240 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.028182 5399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-home\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.029706 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.029637 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-dshm" pod "fa70b62b-4591-401c-bb78-5dfdde976042" (UID: "fa70b62b-4591-401c-bb78-5dfdde976042"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:32:01.029706 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.029671 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa70b62b-4591-401c-bb78-5dfdde976042-kube-api-access-x64th" pod "fa70b62b-4591-401c-bb78-5dfdde976042" (UID: "fa70b62b-4591-401c-bb78-5dfdde976042"). InnerVolumeSpecName "kube-api-access-x64th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:01.029896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.029720 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa70b62b-4591-401c-bb78-5dfdde976042-tls-certs" pod "fa70b62b-4591-401c-bb78-5dfdde976042" (UID: "fa70b62b-4591-401c-bb78-5dfdde976042"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:01.118064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.118022 5399 generic.go:356] "Generic (PLEG): container finished" podID="fa70b62b-4591-401c-bb78-5dfdde976042" containerID="38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8" exitCode=0 Nov 02 20:32:01.118223 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.118068 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" Nov 02 20:32:01.118223 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.118087 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" event={"ID":"fa70b62b-4591-401c-bb78-5dfdde976042","Type":"ContainerDied","Data":"38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8"} Nov 02 20:32:01.118223 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.118126 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68" event={"ID":"fa70b62b-4591-401c-bb78-5dfdde976042","Type":"ContainerDied","Data":"44b47bacd7fee0c74c9a77ff4dbc35a1cd0ad6ead15233f406e8fe3169c42026"} Nov 02 20:32:01.118223 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.118148 5399 scope.go:122] "RemoveContainer" containerID="38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8" Nov 02 20:32:01.129610 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.129421 5399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa70b62b-4591-401c-bb78-5dfdde976042-dshm\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.129610 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.129450 5399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa70b62b-4591-401c-bb78-5dfdde976042-tls-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.129610 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.129462 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x64th\" (UniqueName: \"kubernetes.io/projected/fa70b62b-4591-401c-bb78-5dfdde976042-kube-api-access-x64th\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:01.130689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.130665 5399 scope.go:122] "RemoveContainer" containerID="38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8" Nov 02 20:32:01.131078 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:32:01.131044 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8\": container with ID starting with 38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8 not found: ID does not exist" containerID="38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8" Nov 02 20:32:01.131184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.131085 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8"} err="failed to get container status \"38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8\": rpc error: code = NotFound desc = could not find container \"38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8\": container with ID starting with 38567995d308d76cd2c427150bcf157ecf28575e8eaf631013d0ec6b75386ce8 not found: ID does not exist" Nov 02 20:32:01.236750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.236714 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68"] Nov 02 20:32:01.244347 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.244262 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9fcedd/distinct-2-w2-9fcedd-kserve-557948485f-4zf68"] Nov 02 20:32:01.508561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:01.508479 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" path="/var/lib/kubelet/pods/fa70b62b-4591-401c-bb78-5dfdde976042/volumes" Nov 02 20:32:02.920487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:02.920443 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" podUID="829f41dc-f27c-4d36-9670-eaebac12f098" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:32:02.920885 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:02.920509 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" podUID="e7f94892-433e-4a0e-8a6b-004aadfcab15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:32:12.920125 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:12.920088 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" podUID="829f41dc-f27c-4d36-9670-eaebac12f098" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:32:12.920549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:12.920099 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" podUID="e7f94892-433e-4a0e-8a6b-004aadfcab15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:32:13.964912 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.964881 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s"] Nov 02 20:32:13.965480 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965467 5399 cpu_manager.go:414] "containerMap: removing container" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" Nov 02 20:32:13.965544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965481 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" Nov 02 20:32:13.965544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965509 5399 cpu_manager.go:414] "containerMap: removing container" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" Nov 02 20:32:13.965544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965515 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" Nov 02 20:32:13.965544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965531 5399 cpu_manager.go:414] "containerMap: removing container" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" Nov 02 20:32:13.965544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965536 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" Nov 02 20:32:13.965694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965550 5399 cpu_manager.go:414] "containerMap: removing container" podUID="36a6ac3b-bf3c-4dd5-8844-2397e570117c" containerName="revoke-keys" Nov 02 20:32:13.965694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965555 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a6ac3b-bf3c-4dd5-8844-2397e570117c" containerName="revoke-keys" Nov 02 20:32:13.965694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965635 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a6ac3b-bf3c-4dd5-8844-2397e570117c" containerName="revoke-keys" Nov 02 20:32:13.965694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965644 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa70b62b-4591-401c-bb78-5dfdde976042" containerName="main" Nov 02 20:32:13.965694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965655 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" Nov 02 20:32:13.965694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.965664 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd261a1-a473-4c79-a8d8-71d19923c944" containerName="istio-proxy" Nov 02 20:32:13.969413 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:13.969390 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.024013 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.023982 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s"] Nov 02 20:32:14.065936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.065905 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:14.066077 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.065975 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:32:14.121650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.121611 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hfh\" (UniqueName: \"kubernetes.io/projected/d890279e-519e-436d-96ba-412ffec4eef5-kube-api-access-f6hfh\") pod \"maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.121809 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.121758 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.121809 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.121787 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.223394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.223303 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.223555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.223536 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hfh\" (UniqueName: \"kubernetes.io/projected/d890279e-519e-436d-96ba-412ffec4eef5-kube-api-access-f6hfh\") pod \"maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.223673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.223657 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.223980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.223957 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.225904 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.225883 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.276660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.276633 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hfh\" (UniqueName: \"kubernetes.io/projected/d890279e-519e-436d-96ba-412ffec4eef5-kube-api-access-f6hfh\") pod \"maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.283479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.283449 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:14.668709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:14.668684 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s"] Nov 02 20:32:14.670140 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:32:14.670110 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd890279e_519e_436d_96ba_412ffec4eef5.slice/crio-8445a9f426deafe2445915080e98fed0d767c69960340ccce8715aab17988b6b WatchSource:0}: Error finding container 8445a9f426deafe2445915080e98fed0d767c69960340ccce8715aab17988b6b: Status 404 returned error can't find the container with id 8445a9f426deafe2445915080e98fed0d767c69960340ccce8715aab17988b6b Nov 02 20:32:15.294138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:15.294043 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" event={"ID":"d890279e-519e-436d-96ba-412ffec4eef5","Type":"ContainerStarted","Data":"4d22da0202cfdbb357e4c99e93efbd95b9e49a6ae1593b06e5d1ae93e31882d6"} Nov 02 20:32:15.294138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:15.294098 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" event={"ID":"d890279e-519e-436d-96ba-412ffec4eef5","Type":"ContainerStarted","Data":"8445a9f426deafe2445915080e98fed0d767c69960340ccce8715aab17988b6b"} Nov 02 20:32:15.449552 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:15.449490 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" podStartSLOduration=2.449470298 podStartE2EDuration="2.449470298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:15.403052141 +0000 UTC m=+2378.013704247" watchObservedRunningTime="2026-11-02 20:32:15.449470298 +0000 UTC m=+2378.060122389" Nov 02 20:32:16.306237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:16.306183 5399 generic.go:356] "Generic (PLEG): container finished" podID="d890279e-519e-436d-96ba-412ffec4eef5" containerID="4d22da0202cfdbb357e4c99e93efbd95b9e49a6ae1593b06e5d1ae93e31882d6" exitCode=0 Nov 02 20:32:16.306867 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:16.306399 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" event={"ID":"d890279e-519e-436d-96ba-412ffec4eef5","Type":"ContainerDied","Data":"4d22da0202cfdbb357e4c99e93efbd95b9e49a6ae1593b06e5d1ae93e31882d6"} Nov 02 20:32:17.473631 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:17.473604 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:17.571268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:17.571184 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-openshift-ca\") pod \"d890279e-519e-436d-96ba-412ffec4eef5\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " Nov 02 20:32:17.571463 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:17.571415 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d890279e-519e-436d-96ba-412ffec4eef5-kube-api-access-f6hfh\" (UniqueName: \"kubernetes.io/projected/d890279e-519e-436d-96ba-412ffec4eef5-kube-api-access-f6hfh\") pod \"d890279e-519e-436d-96ba-412ffec4eef5\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " Nov 02 20:32:17.571575 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:17.571558 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-xks-ca\") pod \"d890279e-519e-436d-96ba-412ffec4eef5\" (UID: \"d890279e-519e-436d-96ba-412ffec4eef5\") " Nov 02 20:32:17.571643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:17.571572 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-openshift-ca" pod "d890279e-519e-436d-96ba-412ffec4eef5" (UID: "d890279e-519e-436d-96ba-412ffec4eef5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:17.572042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:17.572017 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:17.573639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:17.573608 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d890279e-519e-436d-96ba-412ffec4eef5-kube-api-access-f6hfh" pod "d890279e-519e-436d-96ba-412ffec4eef5" (UID: "d890279e-519e-436d-96ba-412ffec4eef5"). InnerVolumeSpecName "kube-api-access-f6hfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:17.573752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:17.573653 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-xks-ca" pod "d890279e-519e-436d-96ba-412ffec4eef5" (UID: "d890279e-519e-436d-96ba-412ffec4eef5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:17.673873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:17.673841 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6hfh\" (UniqueName: \"kubernetes.io/projected/d890279e-519e-436d-96ba-412ffec4eef5-kube-api-access-f6hfh\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:17.673873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:17.673871 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d890279e-519e-436d-96ba-412ffec4eef5-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:18.117700 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.117635 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:32:18.236119 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.235883 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:32:18.248643 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:32:18.248581 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:32:18.281871 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.281835 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829f41dc-f27c-4d36-9670-eaebac12f098-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829f41dc-f27c-4d36-9670-eaebac12f098-plugins-config-volume\") pod \"829f41dc-f27c-4d36-9670-eaebac12f098\" (UID: \"829f41dc-f27c-4d36-9670-eaebac12f098\") " Nov 02 20:32:18.282056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.281957 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829f41dc-f27c-4d36-9670-eaebac12f098-kube-api-access-g7s9d\" (UniqueName: \"kubernetes.io/projected/829f41dc-f27c-4d36-9670-eaebac12f098-kube-api-access-g7s9d\") pod \"829f41dc-f27c-4d36-9670-eaebac12f098\" (UID: \"829f41dc-f27c-4d36-9670-eaebac12f098\") " Nov 02 20:32:18.282242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.282199 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829f41dc-f27c-4d36-9670-eaebac12f098-plugins-config-volume" pod "829f41dc-f27c-4d36-9670-eaebac12f098" (UID: "829f41dc-f27c-4d36-9670-eaebac12f098"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:18.282581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.282560 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829f41dc-f27c-4d36-9670-eaebac12f098-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:18.285009 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.284983 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829f41dc-f27c-4d36-9670-eaebac12f098-kube-api-access-g7s9d" pod "829f41dc-f27c-4d36-9670-eaebac12f098" (UID: "829f41dc-f27c-4d36-9670-eaebac12f098"). InnerVolumeSpecName "kube-api-access-g7s9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:18.321794 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.321764 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" Nov 02 20:32:18.321794 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.321776 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s" event={"ID":"d890279e-519e-436d-96ba-412ffec4eef5","Type":"ContainerDied","Data":"8445a9f426deafe2445915080e98fed0d767c69960340ccce8715aab17988b6b"} Nov 02 20:32:18.322028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.321823 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8445a9f426deafe2445915080e98fed0d767c69960340ccce8715aab17988b6b" Nov 02 20:32:18.323387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.323358 5399 generic.go:356] "Generic (PLEG): container finished" podID="829f41dc-f27c-4d36-9670-eaebac12f098" containerID="a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd" exitCode=0 Nov 02 20:32:18.323498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.323392 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" event={"ID":"829f41dc-f27c-4d36-9670-eaebac12f098","Type":"ContainerDied","Data":"a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd"} Nov 02 20:32:18.323498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.323444 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" event={"ID":"829f41dc-f27c-4d36-9670-eaebac12f098","Type":"ContainerDied","Data":"209e4a74b65e176d21e1e397863d2a917d56b5c6e49ef1cfef1a7a678668e368"} Nov 02 20:32:18.323498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.323471 5399 scope.go:122] "RemoveContainer" containerID="a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd" Nov 02 20:32:18.323498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.323401 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm" Nov 02 20:32:18.325236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.325056 5399 generic.go:356] "Generic (PLEG): container finished" podID="e7f94892-433e-4a0e-8a6b-004aadfcab15" containerID="a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef" exitCode=0 Nov 02 20:32:18.325236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.325085 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" event={"ID":"e7f94892-433e-4a0e-8a6b-004aadfcab15","Type":"ContainerDied","Data":"a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef"} Nov 02 20:32:18.325236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.325101 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" Nov 02 20:32:18.325236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.325135 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq" event={"ID":"e7f94892-433e-4a0e-8a6b-004aadfcab15","Type":"ContainerDied","Data":"f67fcac02ed316dfb52ef48ce858acf821fe83f1e1a1fa4b3e07de8673ac10af"} Nov 02 20:32:18.335913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.335896 5399 scope.go:122] "RemoveContainer" containerID="a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd" Nov 02 20:32:18.336197 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:32:18.336180 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd\": container with ID starting with a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd not found: ID does not exist" containerID="a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd" Nov 02 20:32:18.336265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.336204 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd"} err="failed to get container status \"a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd\": rpc error: code = NotFound desc = could not find container \"a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd\": container with ID starting with a8001e4a0e445d147d7f5b8635e8d4818f8751c6ad675c7a6d77eebca13f6fbd not found: ID does not exist" Nov 02 20:32:18.336265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.336221 5399 scope.go:122] "RemoveContainer" containerID="a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef" Nov 02 20:32:18.346624 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.346607 5399 scope.go:122] "RemoveContainer" containerID="a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef" Nov 02 20:32:18.346939 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:32:18.346921 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef\": container with ID starting with a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef not found: ID does not exist" containerID="a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef" Nov 02 20:32:18.347024 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.346945 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef"} err="failed to get container status \"a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef\": rpc error: code = NotFound desc = could not find container \"a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef\": container with ID starting with a4e37d3d0cd3401df4541955a9f3b1c4b5dc5c68cde7f0a7f55c07e3997db6ef not found: ID does not exist" Nov 02 20:32:18.383997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.383913 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f94892-433e-4a0e-8a6b-004aadfcab15-kube-api-access-sz6xn\" (UniqueName: \"kubernetes.io/projected/e7f94892-433e-4a0e-8a6b-004aadfcab15-kube-api-access-sz6xn\") pod \"e7f94892-433e-4a0e-8a6b-004aadfcab15\" (UID: \"e7f94892-433e-4a0e-8a6b-004aadfcab15\") " Nov 02 20:32:18.384144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.384086 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f94892-433e-4a0e-8a6b-004aadfcab15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f94892-433e-4a0e-8a6b-004aadfcab15-plugins-config-volume\") pod \"e7f94892-433e-4a0e-8a6b-004aadfcab15\" (UID: \"e7f94892-433e-4a0e-8a6b-004aadfcab15\") " Nov 02 20:32:18.384420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.384403 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7s9d\" (UniqueName: \"kubernetes.io/projected/829f41dc-f27c-4d36-9670-eaebac12f098-kube-api-access-g7s9d\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:18.384529 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.384462 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f94892-433e-4a0e-8a6b-004aadfcab15-plugins-config-volume" pod "e7f94892-433e-4a0e-8a6b-004aadfcab15" (UID: "e7f94892-433e-4a0e-8a6b-004aadfcab15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:18.386016 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.385994 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f94892-433e-4a0e-8a6b-004aadfcab15-kube-api-access-sz6xn" pod "e7f94892-433e-4a0e-8a6b-004aadfcab15" (UID: "e7f94892-433e-4a0e-8a6b-004aadfcab15"). InnerVolumeSpecName "kube-api-access-sz6xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:18.447228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.447194 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm"] Nov 02 20:32:18.455553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.455524 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9fcedd-b4dd97f8f-vkgpm"] Nov 02 20:32:18.486292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.486247 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz6xn\" (UniqueName: \"kubernetes.io/projected/e7f94892-433e-4a0e-8a6b-004aadfcab15-kube-api-access-sz6xn\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:18.486292 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.486291 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f94892-433e-4a0e-8a6b-004aadfcab15-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:18.741239 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.741060 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq"] Nov 02 20:32:18.744706 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:18.744679 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9fcedd-788bc489dc-lh8zq"] Nov 02 20:32:19.509099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:19.509069 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829f41dc-f27c-4d36-9670-eaebac12f098" path="/var/lib/kubelet/pods/829f41dc-f27c-4d36-9670-eaebac12f098/volumes" Nov 02 20:32:19.509539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:19.509524 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f94892-433e-4a0e-8a6b-004aadfcab15" path="/var/lib/kubelet/pods/e7f94892-433e-4a0e-8a6b-004aadfcab15/volumes" Nov 02 20:32:20.636994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:20.636959 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:32:24.717968 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.717932 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7"] Nov 02 20:32:24.718594 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.718578 5399 cpu_manager.go:414] "containerMap: removing container" podUID="829f41dc-f27c-4d36-9670-eaebac12f098" containerName="payload-processing" Nov 02 20:32:24.718642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.718596 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829f41dc-f27c-4d36-9670-eaebac12f098" containerName="payload-processing" Nov 02 20:32:24.718642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.718608 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d890279e-519e-436d-96ba-412ffec4eef5" containerName="revoke-keys" Nov 02 20:32:24.718642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.718613 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d890279e-519e-436d-96ba-412ffec4eef5" containerName="revoke-keys" Nov 02 20:32:24.718642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.718624 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e7f94892-433e-4a0e-8a6b-004aadfcab15" containerName="payload-pre-processing" Nov 02 20:32:24.718642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.718629 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f94892-433e-4a0e-8a6b-004aadfcab15" containerName="payload-pre-processing" Nov 02 20:32:24.718801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.718740 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="829f41dc-f27c-4d36-9670-eaebac12f098" containerName="payload-processing" Nov 02 20:32:24.718801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.718751 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d890279e-519e-436d-96ba-412ffec4eef5" containerName="revoke-keys" Nov 02 20:32:24.718801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.718758 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f94892-433e-4a0e-8a6b-004aadfcab15" containerName="payload-pre-processing" Nov 02 20:32:24.724058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.724038 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.767852 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.767808 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8spr\" (UniqueName: \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-kube-api-access-v8spr\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.768045 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.767863 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8134e54f-4985-438d-9da1-010c47005ee9-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.768045 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.767996 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.768123 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.768067 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.768166 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.768127 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.768201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.768187 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8134e54f-4985-438d-9da1-010c47005ee9-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.768236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.768217 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.768287 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.768261 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.768383 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.768368 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.772691 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.772661 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9fcedd-openshift-default-dockercfg-rmhls\"" Nov 02 20:32:24.822827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.822786 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7"] Nov 02 20:32:24.831613 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.831580 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r"] Nov 02 20:32:24.835877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.835855 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:24.869812 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.869776 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-819f6ba3-wv96r\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:24.869970 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.869827 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8spr\" (UniqueName: \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-kube-api-access-v8spr\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.869970 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.869889 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcnq\" (UniqueName: \"kubernetes.io/projected/1a641606-742f-4eb3-9dde-bb2f9ab0646a-kube-api-access-8zcnq\") pod \"maas-api-revoke-sub-819f6ba3-wv96r\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:24.869970 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.869930 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8134e54f-4985-438d-9da1-010c47005ee9-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870078 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.869975 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870125 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870078 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870175 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870156 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870195 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8134e54f-4985-438d-9da1-010c47005ee9-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870220 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870316 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-819f6ba3-wv96r\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:24.870371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870348 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870481 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870374 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-credential-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870481 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870410 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870631 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-certs\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870701 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-data\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870720 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-socket\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.870935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.870915 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8134e54f-4985-438d-9da1-010c47005ee9-istiod-ca-cert\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.872509 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.872487 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-envoy\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.872739 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.872718 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8134e54f-4985-438d-9da1-010c47005ee9-istio-podinfo\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.882167 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.882142 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7"] Nov 02 20:32:24.890771 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.890743 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:32:24.893968 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.891504 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:24.896320 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.896301 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:24.919637 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.919604 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r"] Nov 02 20:32:24.922883 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.922851 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-istio-token\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.923022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.922858 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8spr\" (UniqueName: \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-kube-api-access-v8spr\") pod \"e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:24.938379 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.938347 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7"] Nov 02 20:32:24.972172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.972091 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b52de18-75qh7\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:24.972172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.972140 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcnq\" (UniqueName: \"kubernetes.io/projected/1a641606-742f-4eb3-9dde-bb2f9ab0646a-kube-api-access-8zcnq\") pod \"maas-api-revoke-sub-819f6ba3-wv96r\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:24.972451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.972242 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbs8g\" (UniqueName: \"kubernetes.io/projected/115423d0-4be6-4a3c-832e-72c067cd44a8-kube-api-access-rbs8g\") pod \"maas-api-revoke-sub-7b52de18-75qh7\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:24.972512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.972498 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b52de18-75qh7\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:24.972571 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.972547 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-819f6ba3-wv96r\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:24.972748 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.972725 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-819f6ba3-wv96r\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:24.973367 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.973343 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-819f6ba3-wv96r\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:24.974983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:24.974955 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-819f6ba3-wv96r\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:25.025120 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.025088 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcnq\" (UniqueName: \"kubernetes.io/projected/1a641606-742f-4eb3-9dde-bb2f9ab0646a-kube-api-access-8zcnq\") pod \"maas-api-revoke-sub-819f6ba3-wv96r\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:25.038995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.038961 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:25.075306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.075236 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b52de18-75qh7\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:25.075467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.075357 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbs8g\" (UniqueName: \"kubernetes.io/projected/115423d0-4be6-4a3c-832e-72c067cd44a8-kube-api-access-rbs8g\") pod \"maas-api-revoke-sub-7b52de18-75qh7\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:25.075535 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.075491 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b52de18-75qh7\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:25.076015 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.075991 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b52de18-75qh7\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:25.077759 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.077737 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b52de18-75qh7\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:25.128703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.128665 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbs8g\" (UniqueName: \"kubernetes.io/projected/115423d0-4be6-4a3c-832e-72c067cd44a8-kube-api-access-rbs8g\") pod \"maas-api-revoke-sub-7b52de18-75qh7\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:25.150499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.150466 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:25.213727 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.213702 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:25.456873 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:32:25.456835 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8134e54f_4985_438d_9da1_010c47005ee9.slice/crio-c46bed2f78255c1c9db09b4387fb6ba80ec0dc4104e79f59ad51de84a1036ec2 WatchSource:0}: Error finding container c46bed2f78255c1c9db09b4387fb6ba80ec0dc4104e79f59ad51de84a1036ec2: Status 404 returned error can't find the container with id c46bed2f78255c1c9db09b4387fb6ba80ec0dc4104e79f59ad51de84a1036ec2 Nov 02 20:32:25.458351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.458324 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7"] Nov 02 20:32:25.459697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.459667 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:25.459749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.459732 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:25.459785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.459760 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:25.525945 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:32:25.525918 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a641606_742f_4eb3_9dde_bb2f9ab0646a.slice/crio-d7df03ff282864dd720d8c8b191c587e3e1994e0fda18fcf28fd1249b84b79e3 WatchSource:0}: Error finding container d7df03ff282864dd720d8c8b191c587e3e1994e0fda18fcf28fd1249b84b79e3: Status 404 returned error can't find the container with id d7df03ff282864dd720d8c8b191c587e3e1994e0fda18fcf28fd1249b84b79e3 Nov 02 20:32:25.526061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.525982 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r"] Nov 02 20:32:25.604161 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:32:25.604131 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115423d0_4be6_4a3c_832e_72c067cd44a8.slice/crio-50e6949df5a53232bdf3eebef03b5686ae6c4dcce3a2b1bc173af6b7115dd320 WatchSource:0}: Error finding container 50e6949df5a53232bdf3eebef03b5686ae6c4dcce3a2b1bc173af6b7115dd320: Status 404 returned error can't find the container with id 50e6949df5a53232bdf3eebef03b5686ae6c4dcce3a2b1bc173af6b7115dd320 Nov 02 20:32:25.604717 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:25.604612 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7"] Nov 02 20:32:26.392116 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:26.392074 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" event={"ID":"1a641606-742f-4eb3-9dde-bb2f9ab0646a","Type":"ContainerStarted","Data":"ed4d563126e00dc9713c8a7a24b0d3951382414c7744b619bebd044a5bf93ecb"} Nov 02 20:32:26.392653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:26.392140 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" event={"ID":"1a641606-742f-4eb3-9dde-bb2f9ab0646a","Type":"ContainerStarted","Data":"d7df03ff282864dd720d8c8b191c587e3e1994e0fda18fcf28fd1249b84b79e3"} Nov 02 20:32:26.393893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:26.393863 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" event={"ID":"8134e54f-4985-438d-9da1-010c47005ee9","Type":"ContainerStarted","Data":"aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850"} Nov 02 20:32:26.394025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:26.393907 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" event={"ID":"8134e54f-4985-438d-9da1-010c47005ee9","Type":"ContainerStarted","Data":"c46bed2f78255c1c9db09b4387fb6ba80ec0dc4104e79f59ad51de84a1036ec2"} Nov 02 20:32:26.395648 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:26.395624 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" event={"ID":"115423d0-4be6-4a3c-832e-72c067cd44a8","Type":"ContainerStarted","Data":"5fe8148d41f6af5715451f10f83b151af7ed8d2b24f8d2d0ea33847479dacae9"} Nov 02 20:32:26.395738 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:26.395661 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" event={"ID":"115423d0-4be6-4a3c-832e-72c067cd44a8","Type":"ContainerStarted","Data":"50e6949df5a53232bdf3eebef03b5686ae6c4dcce3a2b1bc173af6b7115dd320"} Nov 02 20:32:26.505033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:26.504982 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" podStartSLOduration=2.504969303 podStartE2EDuration="2.504969303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:26.502939878 +0000 UTC m=+2389.113591966" watchObservedRunningTime="2026-11-02 20:32:26.504969303 +0000 UTC m=+2389.115621449" Nov 02 20:32:26.613586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:26.613524 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" podStartSLOduration=2.613505067 podStartE2EDuration="2.613505067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:26.611355774 +0000 UTC m=+2389.222007866" watchObservedRunningTime="2026-11-02 20:32:26.613505067 +0000 UTC m=+2389.224157161" Nov 02 20:32:26.719010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:26.718905 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" podStartSLOduration=2.718890925 podStartE2EDuration="2.718890925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:26.716383818 +0000 UTC m=+2389.327035909" watchObservedRunningTime="2026-11-02 20:32:26.718890925 +0000 UTC m=+2389.329543007" Nov 02 20:32:27.050363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:27.049651 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:27.051856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:27.051825 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:32:27.052011 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:27.051891 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 20:32:27.403939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:27.403904 5399 generic.go:356] "Generic (PLEG): container finished" podID="1a641606-742f-4eb3-9dde-bb2f9ab0646a" containerID="ed4d563126e00dc9713c8a7a24b0d3951382414c7744b619bebd044a5bf93ecb" exitCode=0 Nov 02 20:32:27.404468 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:27.403944 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" event={"ID":"1a641606-742f-4eb3-9dde-bb2f9ab0646a","Type":"ContainerDied","Data":"ed4d563126e00dc9713c8a7a24b0d3951382414c7744b619bebd044a5bf93ecb"} Nov 02 20:32:27.405737 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:27.405714 5399 generic.go:356] "Generic (PLEG): container finished" podID="115423d0-4be6-4a3c-832e-72c067cd44a8" containerID="5fe8148d41f6af5715451f10f83b151af7ed8d2b24f8d2d0ea33847479dacae9" exitCode=0 Nov 02 20:32:27.405869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:27.405797 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" event={"ID":"115423d0-4be6-4a3c-832e-72c067cd44a8","Type":"ContainerDied","Data":"5fe8148d41f6af5715451f10f83b151af7ed8d2b24f8d2d0ea33847479dacae9"} Nov 02 20:32:28.055296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.043843 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:32:28.055296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.043902 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 20:32:28.712617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.712588 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:28.716654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.716633 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:28.869317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.869269 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-openshift-ca\") pod \"115423d0-4be6-4a3c-832e-72c067cd44a8\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " Nov 02 20:32:28.869531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.869345 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a641606-742f-4eb3-9dde-bb2f9ab0646a-kube-api-access-8zcnq\" (UniqueName: \"kubernetes.io/projected/1a641606-742f-4eb3-9dde-bb2f9ab0646a-kube-api-access-8zcnq\") pod \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " Nov 02 20:32:28.869531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.869375 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-openshift-ca\") pod \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " Nov 02 20:32:28.869653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.869532 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-xks-ca\") pod \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\" (UID: \"1a641606-742f-4eb3-9dde-bb2f9ab0646a\") " Nov 02 20:32:28.869653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.869586 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115423d0-4be6-4a3c-832e-72c067cd44a8-kube-api-access-rbs8g\" (UniqueName: \"kubernetes.io/projected/115423d0-4be6-4a3c-832e-72c067cd44a8-kube-api-access-rbs8g\") pod \"115423d0-4be6-4a3c-832e-72c067cd44a8\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " Nov 02 20:32:28.869760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.869679 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-xks-ca\") pod \"115423d0-4be6-4a3c-832e-72c067cd44a8\" (UID: \"115423d0-4be6-4a3c-832e-72c067cd44a8\") " Nov 02 20:32:28.869760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.869685 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-openshift-ca" pod "115423d0-4be6-4a3c-832e-72c067cd44a8" (UID: "115423d0-4be6-4a3c-832e-72c067cd44a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:28.869862 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.869815 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-openshift-ca" pod "1a641606-742f-4eb3-9dde-bb2f9ab0646a" (UID: "1a641606-742f-4eb3-9dde-bb2f9ab0646a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:28.870208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.870180 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:28.870208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.870210 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:28.871877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.871841 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-xks-ca" pod "115423d0-4be6-4a3c-832e-72c067cd44a8" (UID: "115423d0-4be6-4a3c-832e-72c067cd44a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:28.871877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.871870 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-xks-ca" pod "1a641606-742f-4eb3-9dde-bb2f9ab0646a" (UID: "1a641606-742f-4eb3-9dde-bb2f9ab0646a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:28.872432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.872409 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115423d0-4be6-4a3c-832e-72c067cd44a8-kube-api-access-rbs8g" pod "115423d0-4be6-4a3c-832e-72c067cd44a8" (UID: "115423d0-4be6-4a3c-832e-72c067cd44a8"). InnerVolumeSpecName "kube-api-access-rbs8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:28.875757 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.875731 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a641606-742f-4eb3-9dde-bb2f9ab0646a-kube-api-access-8zcnq" pod "1a641606-742f-4eb3-9dde-bb2f9ab0646a" (UID: "1a641606-742f-4eb3-9dde-bb2f9ab0646a"). InnerVolumeSpecName "kube-api-access-8zcnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:28.971952 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.971867 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbs8g\" (UniqueName: \"kubernetes.io/projected/115423d0-4be6-4a3c-832e-72c067cd44a8-kube-api-access-rbs8g\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:28.971952 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.971900 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115423d0-4be6-4a3c-832e-72c067cd44a8-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:28.971952 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.971912 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zcnq\" (UniqueName: \"kubernetes.io/projected/1a641606-742f-4eb3-9dde-bb2f9ab0646a-kube-api-access-8zcnq\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:28.971952 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:28.971922 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a641606-742f-4eb3-9dde-bb2f9ab0646a-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:29.041378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:29.040720 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:32:29.041378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:29.040800 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 20:32:29.424889 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:29.424857 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" Nov 02 20:32:29.425080 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:29.424892 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r" event={"ID":"1a641606-742f-4eb3-9dde-bb2f9ab0646a","Type":"ContainerDied","Data":"d7df03ff282864dd720d8c8b191c587e3e1994e0fda18fcf28fd1249b84b79e3"} Nov 02 20:32:29.425080 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:29.424969 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7df03ff282864dd720d8c8b191c587e3e1994e0fda18fcf28fd1249b84b79e3" Nov 02 20:32:29.449309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:29.436631 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" event={"ID":"115423d0-4be6-4a3c-832e-72c067cd44a8","Type":"ContainerDied","Data":"50e6949df5a53232bdf3eebef03b5686ae6c4dcce3a2b1bc173af6b7115dd320"} Nov 02 20:32:29.449309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:29.436701 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50e6949df5a53232bdf3eebef03b5686ae6c4dcce3a2b1bc173af6b7115dd320" Nov 02 20:32:29.449309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:29.436778 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7" Nov 02 20:32:30.047785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:30.047746 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:32:30.048432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:30.048399 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 20:32:31.048300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:31.039793 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:32:31.048300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:31.039857 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 20:32:32.040089 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:32.040037 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 02 20:32:32.040256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:32.040104 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 02 20:32:33.044396 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:33.044362 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:33.044861 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:33.044820 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:33.045710 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:33.045691 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:32:38.698141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:38.698108 5399 scope.go:122] "RemoveContainer" containerID="478b1cf24bebf8b12c9480cd4441ba777dfd6d0e7f1c4de4d9cc063504ba94e7" Nov 02 20:32:43.098896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.098855 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg"] Nov 02 20:32:43.099702 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.099679 5399 cpu_manager.go:414] "containerMap: removing container" podUID="1a641606-742f-4eb3-9dde-bb2f9ab0646a" containerName="revoke-keys" Nov 02 20:32:43.099802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.099703 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a641606-742f-4eb3-9dde-bb2f9ab0646a" containerName="revoke-keys" Nov 02 20:32:43.099802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.099724 5399 cpu_manager.go:414] "containerMap: removing container" podUID="115423d0-4be6-4a3c-832e-72c067cd44a8" containerName="revoke-keys" Nov 02 20:32:43.099802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.099731 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115423d0-4be6-4a3c-832e-72c067cd44a8" containerName="revoke-keys" Nov 02 20:32:43.099954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.099889 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a641606-742f-4eb3-9dde-bb2f9ab0646a" containerName="revoke-keys" Nov 02 20:32:43.099954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.099904 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="115423d0-4be6-4a3c-832e-72c067cd44a8" containerName="revoke-keys" Nov 02 20:32:43.105558 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.105535 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.158159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.158120 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg"] Nov 02 20:32:43.168196 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.168158 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.168403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.168219 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.168540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.168518 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw68r\" (UniqueName: \"kubernetes.io/projected/7c71726f-17b8-442c-96d3-2d78f3f22ee7-kube-api-access-bw68r\") pod \"maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.197980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.197950 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:43.198141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.197999 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:32:43.269657 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.269618 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw68r\" (UniqueName: \"kubernetes.io/projected/7c71726f-17b8-442c-96d3-2d78f3f22ee7-kube-api-access-bw68r\") pod \"maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.269835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.269729 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.269835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.269754 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.270471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.270441 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.272256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.272228 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.322579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.322543 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw68r\" (UniqueName: \"kubernetes.io/projected/7c71726f-17b8-442c-96d3-2d78f3f22ee7-kube-api-access-bw68r\") pod \"maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.419134 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.419045 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:43.803919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:43.803881 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg"] Nov 02 20:32:43.804432 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:32:43.804400 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c71726f_17b8_442c_96d3_2d78f3f22ee7.slice/crio-a8859f67dd5279293c6901e670636cd3d2c1b6b691f73f08ad8a8e21c101dd07 WatchSource:0}: Error finding container a8859f67dd5279293c6901e670636cd3d2c1b6b691f73f08ad8a8e21c101dd07: Status 404 returned error can't find the container with id a8859f67dd5279293c6901e670636cd3d2c1b6b691f73f08ad8a8e21c101dd07 Nov 02 20:32:44.569856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:44.569817 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" event={"ID":"7c71726f-17b8-442c-96d3-2d78f3f22ee7","Type":"ContainerStarted","Data":"aa485c3d91f36035d120346a219ac6c6a20ed5b0ed4ebdecda98fe38529224c6"} Nov 02 20:32:44.570305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:44.569876 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" event={"ID":"7c71726f-17b8-442c-96d3-2d78f3f22ee7","Type":"ContainerStarted","Data":"a8859f67dd5279293c6901e670636cd3d2c1b6b691f73f08ad8a8e21c101dd07"} Nov 02 20:32:44.676501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:44.676450 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" podStartSLOduration=1.676434416 podStartE2EDuration="1.676434416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:44.675458568 +0000 UTC m=+2407.286110660" watchObservedRunningTime="2026-11-02 20:32:44.676434416 +0000 UTC m=+2407.287086506" Nov 02 20:32:45.577929 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:45.577888 5399 generic.go:356] "Generic (PLEG): container finished" podID="7c71726f-17b8-442c-96d3-2d78f3f22ee7" containerID="aa485c3d91f36035d120346a219ac6c6a20ed5b0ed4ebdecda98fe38529224c6" exitCode=0 Nov 02 20:32:45.578369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:45.577969 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" event={"ID":"7c71726f-17b8-442c-96d3-2d78f3f22ee7","Type":"ContainerDied","Data":"aa485c3d91f36035d120346a219ac6c6a20ed5b0ed4ebdecda98fe38529224c6"} Nov 02 20:32:46.754484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:46.754453 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:46.822083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:46.822046 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c71726f-17b8-442c-96d3-2d78f3f22ee7-kube-api-access-bw68r\" (UniqueName: \"kubernetes.io/projected/7c71726f-17b8-442c-96d3-2d78f3f22ee7-kube-api-access-bw68r\") pod \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " Nov 02 20:32:46.822449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:46.822257 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-openshift-ca\") pod \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " Nov 02 20:32:46.822449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:46.822392 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-xks-ca\") pod \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\" (UID: \"7c71726f-17b8-442c-96d3-2d78f3f22ee7\") " Nov 02 20:32:46.822694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:46.822665 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-openshift-ca" pod "7c71726f-17b8-442c-96d3-2d78f3f22ee7" (UID: "7c71726f-17b8-442c-96d3-2d78f3f22ee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:46.822841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:46.822825 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:46.824420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:46.824394 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-xks-ca" pod "7c71726f-17b8-442c-96d3-2d78f3f22ee7" (UID: "7c71726f-17b8-442c-96d3-2d78f3f22ee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:46.824520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:46.824434 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c71726f-17b8-442c-96d3-2d78f3f22ee7-kube-api-access-bw68r" pod "7c71726f-17b8-442c-96d3-2d78f3f22ee7" (UID: "7c71726f-17b8-442c-96d3-2d78f3f22ee7"). InnerVolumeSpecName "kube-api-access-bw68r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:46.924139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:46.924055 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c71726f-17b8-442c-96d3-2d78f3f22ee7-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:46.924139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:46.924088 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw68r\" (UniqueName: \"kubernetes.io/projected/7c71726f-17b8-442c-96d3-2d78f3f22ee7-kube-api-access-bw68r\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:47.595290 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:47.595256 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" Nov 02 20:32:47.595476 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:47.595270 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg" event={"ID":"7c71726f-17b8-442c-96d3-2d78f3f22ee7","Type":"ContainerDied","Data":"a8859f67dd5279293c6901e670636cd3d2c1b6b691f73f08ad8a8e21c101dd07"} Nov 02 20:32:47.595476 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:47.595345 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8859f67dd5279293c6901e670636cd3d2c1b6b691f73f08ad8a8e21c101dd07" Nov 02 20:32:48.238098 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:32:48.238057 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:32:50.813380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:50.813345 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc"] Nov 02 20:32:50.859679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:50.859640 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c5e8e-jjzvc"] Nov 02 20:32:51.507917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:51.507886 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319d44b8-1f13-41d9-ba98-7222d9575fe3" path="/var/lib/kubelet/pods/319d44b8-1f13-41d9-ba98-7222d9575fe3/volumes" Nov 02 20:32:54.617022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:54.616987 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb"] Nov 02 20:32:54.727100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:54.727061 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5"] Nov 02 20:32:57.601734 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:57.601695 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g"] Nov 02 20:32:57.602154 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:57.601937 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" containerID="cri-o://6bb13825239cad8328bf4083b31c65462a09ba4a20e3a29b02f8f88492e1cd6e" gracePeriod=30 Nov 02 20:32:59.617439 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:59.617382 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" podUID="e7dc89db-90d6-4715-8b34-062ac0779e15" containerName="payload-processing" containerID="cri-o://4f0625dd6d8ec1dd275c37ce71c5b936aee4c9b6eb21e4688a26001ebacf3cff" gracePeriod=25 Nov 02 20:32:59.728474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:59.728432 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" podUID="67e804bb-46a1-4963-af5d-f0fd6a80bdbc" containerName="payload-pre-processing" containerID="cri-o://abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d" gracePeriod=25 Nov 02 20:32:59.758205 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:59.758177 5399 generic.go:356] "Generic (PLEG): container finished" podID="e7dc89db-90d6-4715-8b34-062ac0779e15" containerID="4f0625dd6d8ec1dd275c37ce71c5b936aee4c9b6eb21e4688a26001ebacf3cff" exitCode=0 Nov 02 20:32:59.758353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:59.758256 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" event={"ID":"e7dc89db-90d6-4715-8b34-062ac0779e15","Type":"ContainerDied","Data":"4f0625dd6d8ec1dd275c37ce71c5b936aee4c9b6eb21e4688a26001ebacf3cff"} Nov 02 20:32:59.903699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:32:59.903669 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:33:00.006935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.006885 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7dc89db-90d6-4715-8b34-062ac0779e15-kube-api-access-5zzp9\" (UniqueName: \"kubernetes.io/projected/e7dc89db-90d6-4715-8b34-062ac0779e15-kube-api-access-5zzp9\") pod \"e7dc89db-90d6-4715-8b34-062ac0779e15\" (UID: \"e7dc89db-90d6-4715-8b34-062ac0779e15\") " Nov 02 20:33:00.007151 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.007053 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7dc89db-90d6-4715-8b34-062ac0779e15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7dc89db-90d6-4715-8b34-062ac0779e15-plugins-config-volume\") pod \"e7dc89db-90d6-4715-8b34-062ac0779e15\" (UID: \"e7dc89db-90d6-4715-8b34-062ac0779e15\") " Nov 02 20:33:00.007825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.007775 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7dc89db-90d6-4715-8b34-062ac0779e15-plugins-config-volume" pod "e7dc89db-90d6-4715-8b34-062ac0779e15" (UID: "e7dc89db-90d6-4715-8b34-062ac0779e15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:00.035421 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.035357 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dc89db-90d6-4715-8b34-062ac0779e15-kube-api-access-5zzp9" pod "e7dc89db-90d6-4715-8b34-062ac0779e15" (UID: "e7dc89db-90d6-4715-8b34-062ac0779e15"). InnerVolumeSpecName "kube-api-access-5zzp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:00.061896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.061870 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:33:00.108999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.108963 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7dc89db-90d6-4715-8b34-062ac0779e15-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:00.108999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.108990 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zzp9\" (UniqueName: \"kubernetes.io/projected/e7dc89db-90d6-4715-8b34-062ac0779e15-kube-api-access-5zzp9\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:00.209625 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.209537 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-kube-api-access-wnsvc\" (UniqueName: \"kubernetes.io/projected/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-kube-api-access-wnsvc\") pod \"67e804bb-46a1-4963-af5d-f0fd6a80bdbc\" (UID: \"67e804bb-46a1-4963-af5d-f0fd6a80bdbc\") " Nov 02 20:33:00.209801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.209757 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-plugins-config-volume\") pod \"67e804bb-46a1-4963-af5d-f0fd6a80bdbc\" (UID: \"67e804bb-46a1-4963-af5d-f0fd6a80bdbc\") " Nov 02 20:33:00.210160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.210133 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-plugins-config-volume" pod "67e804bb-46a1-4963-af5d-f0fd6a80bdbc" (UID: "67e804bb-46a1-4963-af5d-f0fd6a80bdbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:00.210342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.210322 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:00.211642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.211621 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-kube-api-access-wnsvc" pod "67e804bb-46a1-4963-af5d-f0fd6a80bdbc" (UID: "67e804bb-46a1-4963-af5d-f0fd6a80bdbc"). InnerVolumeSpecName "kube-api-access-wnsvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:00.311625 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.311590 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnsvc\" (UniqueName: \"kubernetes.io/projected/67e804bb-46a1-4963-af5d-f0fd6a80bdbc-kube-api-access-wnsvc\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:00.767095 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.767060 5399 generic.go:356] "Generic (PLEG): container finished" podID="67e804bb-46a1-4963-af5d-f0fd6a80bdbc" containerID="abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d" exitCode=0 Nov 02 20:33:00.767618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.767111 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" Nov 02 20:33:00.767618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.767147 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" event={"ID":"67e804bb-46a1-4963-af5d-f0fd6a80bdbc","Type":"ContainerDied","Data":"abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d"} Nov 02 20:33:00.767618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.767209 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5" event={"ID":"67e804bb-46a1-4963-af5d-f0fd6a80bdbc","Type":"ContainerDied","Data":"1d5cd9c0a3ca8f3af6b0e02fb80fac40ad751a32abe1697b01665310ba58fc4b"} Nov 02 20:33:00.767618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.767232 5399 scope.go:122] "RemoveContainer" containerID="abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d" Nov 02 20:33:00.768996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.768978 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" event={"ID":"e7dc89db-90d6-4715-8b34-062ac0779e15","Type":"ContainerDied","Data":"4cc48aa19561708fa0c432ecc74e5f7a49360eecb9a19cbf4c90cdfe55966db8"} Nov 02 20:33:00.769099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.769027 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb" Nov 02 20:33:00.784676 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.784461 5399 scope.go:122] "RemoveContainer" containerID="abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d" Nov 02 20:33:00.791672 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:33:00.791641 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d\": container with ID starting with abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d not found: ID does not exist" containerID="abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d" Nov 02 20:33:00.791786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.791684 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d"} err="failed to get container status \"abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d\": rpc error: code = NotFound desc = could not find container \"abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d\": container with ID starting with abe560ac8354ebb9a78ee589d4cf37cc03efd142941230942cd34836e9b7707d not found: ID does not exist" Nov 02 20:33:00.791786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.791712 5399 scope.go:122] "RemoveContainer" containerID="4f0625dd6d8ec1dd275c37ce71c5b936aee4c9b6eb21e4688a26001ebacf3cff" Nov 02 20:33:00.921271 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.921185 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5"] Nov 02 20:33:00.967070 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:00.967037 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-62d287-669b665775-4c2w5"] Nov 02 20:33:01.024424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:01.024390 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb"] Nov 02 20:33:01.027703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:01.027676 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-62d287-86cb796d99-ckgbb"] Nov 02 20:33:01.508897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:01.508863 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e804bb-46a1-4963-af5d-f0fd6a80bdbc" path="/var/lib/kubelet/pods/67e804bb-46a1-4963-af5d-f0fd6a80bdbc/volumes" Nov 02 20:33:01.509355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:01.509340 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7dc89db-90d6-4715-8b34-062ac0779e15" path="/var/lib/kubelet/pods/e7dc89db-90d6-4715-8b34-062ac0779e15/volumes" Nov 02 20:33:02.269951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:02.269913 5399 patch_prober.go:29] interesting pod/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:33:02.270373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:02.269967 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:33:02.792183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:02.792151 5399 generic.go:356] "Generic (PLEG): container finished" podID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerID="6bb13825239cad8328bf4083b31c65462a09ba4a20e3a29b02f8f88492e1cd6e" exitCode=0 Nov 02 20:33:02.792439 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:02.792227 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" event={"ID":"81a960d8-7c24-462a-9a0d-e1205a4ee579","Type":"ContainerDied","Data":"6bb13825239cad8328bf4083b31c65462a09ba4a20e3a29b02f8f88492e1cd6e"} Nov 02 20:33:02.792439 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:02.792326 5399 scope.go:122] "RemoveContainer" containerID="ddfa6626bc66e0a91a179e1cf437117f535630e3439013fac48a44691d092886" Nov 02 20:33:02.869705 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:02.869638 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:33:03.049769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.049735 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-token\" (UniqueName: \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-token\") pod \"81a960d8-7c24-462a-9a0d-e1205a4ee579\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " Nov 02 20:33:03.049944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.049782 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-credential-socket\") pod \"81a960d8-7c24-462a-9a0d-e1205a4ee579\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " Nov 02 20:33:03.049944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.049883 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a960d8-7c24-462a-9a0d-e1205a4ee579-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a960d8-7c24-462a-9a0d-e1205a4ee579-istiod-ca-cert\") pod \"81a960d8-7c24-462a-9a0d-e1205a4ee579\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " Nov 02 20:33:03.049944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.049921 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-podinfo\") pod \"81a960d8-7c24-462a-9a0d-e1205a4ee579\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " Nov 02 20:33:03.050139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.049946 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-kube-api-access-fjqll\" (UniqueName: \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-kube-api-access-fjqll\") pod \"81a960d8-7c24-462a-9a0d-e1205a4ee579\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " Nov 02 20:33:03.050139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050023 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-envoy\") pod \"81a960d8-7c24-462a-9a0d-e1205a4ee579\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " Nov 02 20:33:03.050139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050123 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-certs\") pod \"81a960d8-7c24-462a-9a0d-e1205a4ee579\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " Nov 02 20:33:03.050357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050183 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-socket\") pod \"81a960d8-7c24-462a-9a0d-e1205a4ee579\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " Nov 02 20:33:03.050357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050180 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-credential-socket" pod "81a960d8-7c24-462a-9a0d-e1205a4ee579" (UID: "81a960d8-7c24-462a-9a0d-e1205a4ee579"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:03.050357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050249 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-data\") pod \"81a960d8-7c24-462a-9a0d-e1205a4ee579\" (UID: \"81a960d8-7c24-462a-9a0d-e1205a4ee579\") " Nov 02 20:33:03.050523 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050442 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-certs" pod "81a960d8-7c24-462a-9a0d-e1205a4ee579" (UID: "81a960d8-7c24-462a-9a0d-e1205a4ee579"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:03.050667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050644 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-data" pod "81a960d8-7c24-462a-9a0d-e1205a4ee579" (UID: "81a960d8-7c24-462a-9a0d-e1205a4ee579"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:03.050731 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050654 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-socket" pod "81a960d8-7c24-462a-9a0d-e1205a4ee579" (UID: "81a960d8-7c24-462a-9a0d-e1205a4ee579"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:03.050799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050726 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a960d8-7c24-462a-9a0d-e1205a4ee579-istiod-ca-cert" pod "81a960d8-7c24-462a-9a0d-e1205a4ee579" (UID: "81a960d8-7c24-462a-9a0d-e1205a4ee579"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:03.050968 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050948 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a960d8-7c24-462a-9a0d-e1205a4ee579-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.051037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050973 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.051037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.050989 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.051037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.051003 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.051037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.051016 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.052714 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.052439 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-token" pod "81a960d8-7c24-462a-9a0d-e1205a4ee579" (UID: "81a960d8-7c24-462a-9a0d-e1205a4ee579"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:03.053560 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.053528 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-envoy" pod "81a960d8-7c24-462a-9a0d-e1205a4ee579" (UID: "81a960d8-7c24-462a-9a0d-e1205a4ee579"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:03.053887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.053866 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-podinfo" pod "81a960d8-7c24-462a-9a0d-e1205a4ee579" (UID: "81a960d8-7c24-462a-9a0d-e1205a4ee579"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:33:03.054102 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.054073 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-kube-api-access-fjqll" pod "81a960d8-7c24-462a-9a0d-e1205a4ee579" (UID: "81a960d8-7c24-462a-9a0d-e1205a4ee579"). InnerVolumeSpecName "kube-api-access-fjqll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:03.151983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.151900 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.151983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.151928 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjqll\" (UniqueName: \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-kube-api-access-fjqll\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.151983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.151938 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.151983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.151947 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a960d8-7c24-462a-9a0d-e1205a4ee579-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.800518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.800403 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" Nov 02 20:33:03.800938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.800410 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g" event={"ID":"81a960d8-7c24-462a-9a0d-e1205a4ee579","Type":"ContainerDied","Data":"b67bbc4bec1c6798316b2624039de36d1942f04e5ee8b42e1dc2e94585a66466"} Nov 02 20:33:03.800938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.800614 5399 scope.go:122] "RemoveContainer" containerID="6bb13825239cad8328bf4083b31c65462a09ba4a20e3a29b02f8f88492e1cd6e" Nov 02 20:33:03.965021 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.964984 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g"] Nov 02 20:33:03.969259 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:03.969228 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8c5e8e-openshift-default-7f88888bc9-w2m9g"] Nov 02 20:33:05.514619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:05.514587 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" path="/var/lib/kubelet/pods/81a960d8-7c24-462a-9a0d-e1205a4ee579/volumes" Nov 02 20:33:07.346596 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:07.346554 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:33:09.040823 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.040791 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4"] Nov 02 20:33:09.041392 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041367 5399 cpu_manager.go:414] "containerMap: removing container" podUID="67e804bb-46a1-4963-af5d-f0fd6a80bdbc" containerName="payload-pre-processing" Nov 02 20:33:09.041392 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041383 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e804bb-46a1-4963-af5d-f0fd6a80bdbc" containerName="payload-pre-processing" Nov 02 20:33:09.041483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041396 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7c71726f-17b8-442c-96d3-2d78f3f22ee7" containerName="revoke-keys" Nov 02 20:33:09.041483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041401 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c71726f-17b8-442c-96d3-2d78f3f22ee7" containerName="revoke-keys" Nov 02 20:33:09.041483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041424 5399 cpu_manager.go:414] "containerMap: removing container" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" Nov 02 20:33:09.041483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041429 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" Nov 02 20:33:09.041483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041455 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e7dc89db-90d6-4715-8b34-062ac0779e15" containerName="payload-processing" Nov 02 20:33:09.041483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041459 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7dc89db-90d6-4715-8b34-062ac0779e15" containerName="payload-processing" Nov 02 20:33:09.041701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041558 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" Nov 02 20:33:09.041701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041567 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7dc89db-90d6-4715-8b34-062ac0779e15" containerName="payload-processing" Nov 02 20:33:09.041701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041577 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e804bb-46a1-4963-af5d-f0fd6a80bdbc" containerName="payload-pre-processing" Nov 02 20:33:09.041701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041583 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c71726f-17b8-442c-96d3-2d78f3f22ee7" containerName="revoke-keys" Nov 02 20:33:09.041701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.041592 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" Nov 02 20:33:09.046985 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.046965 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:33:09.096696 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.096665 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4"] Nov 02 20:33:09.129775 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.129740 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4\" (UID: \"4b0f654d-dc58-4977-a3a0-b7d35b280f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:33:09.130010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.129822 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rxw\" (UniqueName: \"kubernetes.io/projected/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-kube-api-access-v2rxw\") pod \"payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4\" (UID: \"4b0f654d-dc58-4977-a3a0-b7d35b280f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:33:09.207343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.157411 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0d9ec0-dockercfg-crvzv\"" Nov 02 20:33:09.207343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.180459 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0d9ec0\"" Nov 02 20:33:09.238841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.231759 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rxw\" (UniqueName: \"kubernetes.io/projected/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-kube-api-access-v2rxw\") pod \"payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4\" (UID: \"4b0f654d-dc58-4977-a3a0-b7d35b280f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:33:09.238841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.231998 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4\" (UID: \"4b0f654d-dc58-4977-a3a0-b7d35b280f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:33:09.238841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.235165 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4\" (UID: \"4b0f654d-dc58-4977-a3a0-b7d35b280f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:33:09.288358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.288321 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rxw\" (UniqueName: \"kubernetes.io/projected/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-kube-api-access-v2rxw\") pod \"payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4\" (UID: \"4b0f654d-dc58-4977-a3a0-b7d35b280f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:33:09.361032 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.361000 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:33:09.728701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.728656 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4"] Nov 02 20:33:09.730800 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:33:09.730768 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0f654d_dc58_4977_a3a0_b7d35b280f7c.slice/crio-a808f8f2e4b77f75606c0e28a5c79d2c42edec2e0ac7e768c8526e230bae2fff WatchSource:0}: Error finding container a808f8f2e4b77f75606c0e28a5c79d2c42edec2e0ac7e768c8526e230bae2fff: Status 404 returned error can't find the container with id a808f8f2e4b77f75606c0e28a5c79d2c42edec2e0ac7e768c8526e230bae2fff Nov 02 20:33:09.854479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.854446 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" event={"ID":"4b0f654d-dc58-4977-a3a0-b7d35b280f7c","Type":"ContainerStarted","Data":"c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94"} Nov 02 20:33:09.854647 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.854522 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" event={"ID":"4b0f654d-dc58-4977-a3a0-b7d35b280f7c","Type":"ContainerStarted","Data":"a808f8f2e4b77f75606c0e28a5c79d2c42edec2e0ac7e768c8526e230bae2fff"} Nov 02 20:33:09.854705 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.854666 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:33:09.974726 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:09.974634 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" podStartSLOduration=0.974615555 podStartE2EDuration="974.615555ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:09.969020579 +0000 UTC m=+2432.579672670" watchObservedRunningTime="2026-11-02 20:33:09.974615555 +0000 UTC m=+2432.585267648" Nov 02 20:33:18.277371 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:33:18.277228 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:33:20.847836 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:20.847798 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd"] Nov 02 20:33:20.865552 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:20.865521 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:33:20.879449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:20.879411 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a51b3f-wwbsd"] Nov 02 20:33:21.509376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:21.509345 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a6ac3b-bf3c-4dd5-8844-2397e570117c" path="/var/lib/kubelet/pods/36a6ac3b-bf3c-4dd5-8844-2397e570117c/volumes" Nov 02 20:33:26.938587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:26.938548 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s"] Nov 02 20:33:26.984485 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:26.984446 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ab501-lsl9s"] Nov 02 20:33:27.163415 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:27.163382 5399 patch_prober.go:29] interesting pod/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:33:27.163595 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:27.163443 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:33:27.512829 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:27.512784 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d890279e-519e-436d-96ba-412ffec4eef5" path="/var/lib/kubelet/pods/d890279e-519e-436d-96ba-412ffec4eef5/volumes" Nov 02 20:33:29.028493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.028435 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl"] Nov 02 20:33:29.029936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.029906 5399 cpu_manager.go:414] "containerMap: removing container" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" Nov 02 20:33:29.029936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.029933 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a960d8-7c24-462a-9a0d-e1205a4ee579" containerName="istio-proxy" Nov 02 20:33:29.034324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.034300 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.050691 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.050661 5399 generic.go:356] "Generic (PLEG): container finished" podID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerID="a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca" exitCode=0 Nov 02 20:33:29.050816 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.050733 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" event={"ID":"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f","Type":"ContainerDied","Data":"a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca"} Nov 02 20:33:29.051157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.051142 5399 scope.go:122] "RemoveContainer" containerID="a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca" Nov 02 20:33:29.052755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.052726 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:29.052858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.052791 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:29.052858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.052822 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:29.077914 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.077882 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdsl2\" (UniqueName: \"kubernetes.io/projected/33890af3-f7cb-4a17-99aa-24ed74d901cb-kube-api-access-pdsl2\") pod \"maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.078106 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.078082 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.078241 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.078213 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.084621 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.084593 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl"] Nov 02 20:33:29.126467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.126437 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:33:29.126811 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.126785 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:33:29.179662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.179626 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdsl2\" (UniqueName: \"kubernetes.io/projected/33890af3-f7cb-4a17-99aa-24ed74d901cb-kube-api-access-pdsl2\") pod \"maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.179851 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.179782 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.179924 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.179860 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.180556 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.180532 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.182270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.182248 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.233041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.233013 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdsl2\" (UniqueName: \"kubernetes.io/projected/33890af3-f7cb-4a17-99aa-24ed74d901cb-kube-api-access-pdsl2\") pod \"maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.351051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.351013 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:29.749465 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:29.749434 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl"] Nov 02 20:33:29.749739 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:33:29.749708 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33890af3_f7cb_4a17_99aa_24ed74d901cb.slice/crio-4e99e62bfd0ca134ff75f9e5d6aa1f161afc253404a248bca55dda5c045b06e1 WatchSource:0}: Error finding container 4e99e62bfd0ca134ff75f9e5d6aa1f161afc253404a248bca55dda5c045b06e1: Status 404 returned error can't find the container with id 4e99e62bfd0ca134ff75f9e5d6aa1f161afc253404a248bca55dda5c045b06e1 Nov 02 20:33:30.069154 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:30.069115 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" event={"ID":"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f","Type":"ContainerStarted","Data":"470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab"} Nov 02 20:33:30.070808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:30.070780 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" event={"ID":"33890af3-f7cb-4a17-99aa-24ed74d901cb","Type":"ContainerStarted","Data":"4e99e62bfd0ca134ff75f9e5d6aa1f161afc253404a248bca55dda5c045b06e1"} Nov 02 20:33:30.778246 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:30.778213 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:33:30.778982 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:30.778956 5399 patch_prober.go:29] interesting pod/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:33:30.779101 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:30.779003 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 20:33:31.084181 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:31.084144 5399 generic.go:356] "Generic (PLEG): container finished" podID="33890af3-f7cb-4a17-99aa-24ed74d901cb" containerID="8368aaa891deeff6641a40351e3631345f3457a013f90fd3c8851aea73b40b32" exitCode=0 Nov 02 20:33:31.084675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:31.084224 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" event={"ID":"33890af3-f7cb-4a17-99aa-24ed74d901cb","Type":"ContainerDied","Data":"8368aaa891deeff6641a40351e3631345f3457a013f90fd3c8851aea73b40b32"} Nov 02 20:33:31.778128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:31.778092 5399 patch_prober.go:29] interesting pod/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:33:31.778359 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:31.778149 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 20:33:32.445899 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.445863 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:32.500334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.500267 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33890af3-f7cb-4a17-99aa-24ed74d901cb-kube-api-access-pdsl2\" (UniqueName: \"kubernetes.io/projected/33890af3-f7cb-4a17-99aa-24ed74d901cb-kube-api-access-pdsl2\") pod \"33890af3-f7cb-4a17-99aa-24ed74d901cb\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " Nov 02 20:33:32.508297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.503576 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-xks-ca\") pod \"33890af3-f7cb-4a17-99aa-24ed74d901cb\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " Nov 02 20:33:32.508297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.503781 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-openshift-ca\") pod \"33890af3-f7cb-4a17-99aa-24ed74d901cb\" (UID: \"33890af3-f7cb-4a17-99aa-24ed74d901cb\") " Nov 02 20:33:32.508297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.504709 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-openshift-ca" pod "33890af3-f7cb-4a17-99aa-24ed74d901cb" (UID: "33890af3-f7cb-4a17-99aa-24ed74d901cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:32.518342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.510863 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33890af3-f7cb-4a17-99aa-24ed74d901cb-kube-api-access-pdsl2" pod "33890af3-f7cb-4a17-99aa-24ed74d901cb" (UID: "33890af3-f7cb-4a17-99aa-24ed74d901cb"). InnerVolumeSpecName "kube-api-access-pdsl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:32.521633 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.521590 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-xks-ca" pod "33890af3-f7cb-4a17-99aa-24ed74d901cb" (UID: "33890af3-f7cb-4a17-99aa-24ed74d901cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:32.605788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.605753 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:32.605788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.605791 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33890af3-f7cb-4a17-99aa-24ed74d901cb-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:32.605994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.605808 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdsl2\" (UniqueName: \"kubernetes.io/projected/33890af3-f7cb-4a17-99aa-24ed74d901cb-kube-api-access-pdsl2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:32.778208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.778130 5399 patch_prober.go:29] interesting pod/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:33:32.778208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:32.778198 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 20:33:33.086301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:33.086238 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb"] Nov 02 20:33:33.096516 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:33.096484 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f"] Nov 02 20:33:33.118010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:33.117975 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" event={"ID":"33890af3-f7cb-4a17-99aa-24ed74d901cb","Type":"ContainerDied","Data":"4e99e62bfd0ca134ff75f9e5d6aa1f161afc253404a248bca55dda5c045b06e1"} Nov 02 20:33:33.118172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:33.118015 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl" Nov 02 20:33:33.118172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:33.118033 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e99e62bfd0ca134ff75f9e5d6aa1f161afc253404a248bca55dda5c045b06e1" Nov 02 20:33:33.765607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:33.765570 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p"] Nov 02 20:33:33.766255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:33.765875 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" containerID="cri-o://335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5" gracePeriod=30 Nov 02 20:33:33.778586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:33.778556 5399 patch_prober.go:29] interesting pod/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:33:33.778751 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:33.778605 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 20:33:34.778411 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:34.778378 5399 patch_prober.go:29] interesting pod/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:33:34.778815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:34.778434 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 02 20:33:35.783513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:35.783481 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:33:35.783992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:35.783895 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:33:35.784783 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:35.784762 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:33:35.991451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:35.991411 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm"] Nov 02 20:33:35.992197 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:35.992171 5399 cpu_manager.go:414] "containerMap: removing container" podUID="33890af3-f7cb-4a17-99aa-24ed74d901cb" containerName="revoke-keys" Nov 02 20:33:35.992375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:35.992352 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33890af3-f7cb-4a17-99aa-24ed74d901cb" containerName="revoke-keys" Nov 02 20:33:35.992515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:35.992504 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="33890af3-f7cb-4a17-99aa-24ed74d901cb" containerName="revoke-keys" Nov 02 20:33:35.996211 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:35.996194 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.050630 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.050538 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:33:36.050812 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.050640 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:33:36.064626 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.064574 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm"] Nov 02 20:33:36.068168 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.068138 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.068360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.068266 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.068442 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.068421 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrwk\" (UniqueName: \"kubernetes.io/projected/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-kube-api-access-hcrwk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.169957 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.169925 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.170172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.170015 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.170172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.170109 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrwk\" (UniqueName: \"kubernetes.io/projected/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-kube-api-access-hcrwk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.170706 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.170680 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.172557 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.172529 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.230566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.230533 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrwk\" (UniqueName: \"kubernetes.io/projected/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-kube-api-access-hcrwk\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.311170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.311140 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:36.748878 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:36.748848 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm"] Nov 02 20:33:36.749400 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:33:36.749374 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7640641e_c713_4a9f_ab1e_0ee19b6f02aa.slice/crio-3ae573a842ebb373dcab409746c178b77a3cbb614d72f8da8adac0ce512757c6 WatchSource:0}: Error finding container 3ae573a842ebb373dcab409746c178b77a3cbb614d72f8da8adac0ce512757c6: Status 404 returned error can't find the container with id 3ae573a842ebb373dcab409746c178b77a3cbb614d72f8da8adac0ce512757c6 Nov 02 20:33:37.172678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:37.172645 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" event={"ID":"7640641e-c713-4a9f-ab1e-0ee19b6f02aa","Type":"ContainerStarted","Data":"f69fa82bfa0e7590b42ad742ec4532cc0ad90bb4032e611e5f6d7e6dfefa6f51"} Nov 02 20:33:37.173166 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:37.172700 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" event={"ID":"7640641e-c713-4a9f-ab1e-0ee19b6f02aa","Type":"ContainerStarted","Data":"3ae573a842ebb373dcab409746c178b77a3cbb614d72f8da8adac0ce512757c6"} Nov 02 20:33:37.283851 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:37.283791 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" podStartSLOduration=2.283776002 podStartE2EDuration="2.283776002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:37.280944072 +0000 UTC m=+2459.891596168" watchObservedRunningTime="2026-11-02 20:33:37.283776002 +0000 UTC m=+2459.894428093" Nov 02 20:33:38.050450 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:38.050217 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g"] Nov 02 20:33:38.053779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:38.053750 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9fcedd-jhk6g"] Nov 02 20:33:38.182807 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:38.182767 5399 generic.go:356] "Generic (PLEG): container finished" podID="7640641e-c713-4a9f-ab1e-0ee19b6f02aa" containerID="f69fa82bfa0e7590b42ad742ec4532cc0ad90bb4032e611e5f6d7e6dfefa6f51" exitCode=0 Nov 02 20:33:38.183201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:38.182851 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" event={"ID":"7640641e-c713-4a9f-ab1e-0ee19b6f02aa","Type":"ContainerDied","Data":"f69fa82bfa0e7590b42ad742ec4532cc0ad90bb4032e611e5f6d7e6dfefa6f51"} Nov 02 20:33:39.042427 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.042403 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:33:39.156378 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.156340 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-credential-socket\") pod \"32778100-8a2b-46c9-b507-2273c72e10af\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " Nov 02 20:33:39.156587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.156422 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-data\") pod \"32778100-8a2b-46c9-b507-2273c72e10af\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " Nov 02 20:33:39.156587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.156476 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-envoy\") pod \"32778100-8a2b-46c9-b507-2273c72e10af\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " Nov 02 20:33:39.156587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.156558 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-socket\") pod \"32778100-8a2b-46c9-b507-2273c72e10af\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " Nov 02 20:33:39.156779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.156602 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-kube-api-access-9zxvj\" (UniqueName: \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-kube-api-access-9zxvj\") pod \"32778100-8a2b-46c9-b507-2273c72e10af\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " Nov 02 20:33:39.156779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.156663 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32778100-8a2b-46c9-b507-2273c72e10af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32778100-8a2b-46c9-b507-2273c72e10af-istiod-ca-cert\") pod \"32778100-8a2b-46c9-b507-2273c72e10af\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " Nov 02 20:33:39.156779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.156722 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-istio-token\" (UniqueName: \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-istio-token\") pod \"32778100-8a2b-46c9-b507-2273c72e10af\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " Nov 02 20:33:39.156938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.156803 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32778100-8a2b-46c9-b507-2273c72e10af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32778100-8a2b-46c9-b507-2273c72e10af-istio-podinfo\") pod \"32778100-8a2b-46c9-b507-2273c72e10af\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " Nov 02 20:33:39.156938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.156863 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-certs\") pod \"32778100-8a2b-46c9-b507-2273c72e10af\" (UID: \"32778100-8a2b-46c9-b507-2273c72e10af\") " Nov 02 20:33:39.157846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.157696 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-socket" pod "32778100-8a2b-46c9-b507-2273c72e10af" (UID: "32778100-8a2b-46c9-b507-2273c72e10af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:39.157972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.157899 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32778100-8a2b-46c9-b507-2273c72e10af-istiod-ca-cert" pod "32778100-8a2b-46c9-b507-2273c72e10af" (UID: "32778100-8a2b-46c9-b507-2273c72e10af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:39.158343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.158085 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-data" pod "32778100-8a2b-46c9-b507-2273c72e10af" (UID: "32778100-8a2b-46c9-b507-2273c72e10af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:39.158343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.158199 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-certs" pod "32778100-8a2b-46c9-b507-2273c72e10af" (UID: "32778100-8a2b-46c9-b507-2273c72e10af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:39.158583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.158545 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-credential-socket" pod "32778100-8a2b-46c9-b507-2273c72e10af" (UID: "32778100-8a2b-46c9-b507-2273c72e10af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:39.160344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.160318 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-envoy" pod "32778100-8a2b-46c9-b507-2273c72e10af" (UID: "32778100-8a2b-46c9-b507-2273c72e10af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:39.160637 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.160613 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-kube-api-access-9zxvj" pod "32778100-8a2b-46c9-b507-2273c72e10af" (UID: "32778100-8a2b-46c9-b507-2273c72e10af"). InnerVolumeSpecName "kube-api-access-9zxvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:39.160953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.160909 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-istio-token" pod "32778100-8a2b-46c9-b507-2273c72e10af" (UID: "32778100-8a2b-46c9-b507-2273c72e10af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:39.161870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.161845 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32778100-8a2b-46c9-b507-2273c72e10af-istio-podinfo" pod "32778100-8a2b-46c9-b507-2273c72e10af" (UID: "32778100-8a2b-46c9-b507-2273c72e10af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:33:39.192456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.192419 5399 generic.go:356] "Generic (PLEG): container finished" podID="32778100-8a2b-46c9-b507-2273c72e10af" containerID="335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5" exitCode=0 Nov 02 20:33:39.192870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.192471 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" Nov 02 20:33:39.192870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.192503 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" event={"ID":"32778100-8a2b-46c9-b507-2273c72e10af","Type":"ContainerDied","Data":"335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5"} Nov 02 20:33:39.192870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.192605 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p" event={"ID":"32778100-8a2b-46c9-b507-2273c72e10af","Type":"ContainerDied","Data":"15139c6641c27ef705d7f3762dec015abd94866bb56e4de79ec5fc37ed8130b5"} Nov 02 20:33:39.192870 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.192642 5399 scope.go:122] "RemoveContainer" containerID="335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5" Nov 02 20:33:39.220694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.220659 5399 scope.go:122] "RemoveContainer" containerID="699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57" Nov 02 20:33:39.231980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.231954 5399 scope.go:122] "RemoveContainer" containerID="335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5" Nov 02 20:33:39.232784 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:33:39.232260 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5\": container with ID starting with 335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5 not found: ID does not exist" containerID="335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5" Nov 02 20:33:39.232784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.232330 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5"} err="failed to get container status \"335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5\": rpc error: code = NotFound desc = could not find container \"335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5\": container with ID starting with 335b0fc780b47b1387725ce9ebf5918a421bd76554176e2a3a33a15ab74249d5 not found: ID does not exist" Nov 02 20:33:39.232784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.232349 5399 scope.go:122] "RemoveContainer" containerID="699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57" Nov 02 20:33:39.232784 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:33:39.232573 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57\": container with ID starting with 699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57 not found: ID does not exist" containerID="699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57" Nov 02 20:33:39.232784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.232592 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57"} err="failed to get container status \"699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57\": rpc error: code = NotFound desc = could not find container \"699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57\": container with ID starting with 699e492edbcc377fff827243963b7e6c1115b17be17dc0bc5990ff605edcdf57 not found: ID does not exist" Nov 02 20:33:39.259188 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.259154 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.261530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.259190 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.261530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.259204 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.261530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.259215 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.261530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.259227 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32778100-8a2b-46c9-b507-2273c72e10af-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.261530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.259241 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zxvj\" (UniqueName: \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-kube-api-access-9zxvj\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.261530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.259257 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32778100-8a2b-46c9-b507-2273c72e10af-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.261530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.259288 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32778100-8a2b-46c9-b507-2273c72e10af-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.261530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.259303 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32778100-8a2b-46c9-b507-2273c72e10af-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.348003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.347970 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p"] Nov 02 20:33:39.350504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.350486 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:39.351466 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.351444 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1ab501-openshift-default-68bdbd4475-rfg4p"] Nov 02 20:33:39.461009 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.460962 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-kube-api-access-hcrwk\" (UniqueName: \"kubernetes.io/projected/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-kube-api-access-hcrwk\") pod \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " Nov 02 20:33:39.461199 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.461180 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-openshift-ca\") pod \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " Nov 02 20:33:39.461290 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.461227 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-xks-ca\") pod \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\" (UID: \"7640641e-c713-4a9f-ab1e-0ee19b6f02aa\") " Nov 02 20:33:39.461581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.461556 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-openshift-ca" pod "7640641e-c713-4a9f-ab1e-0ee19b6f02aa" (UID: "7640641e-c713-4a9f-ab1e-0ee19b6f02aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:39.461736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.461721 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.463399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.463371 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-xks-ca" pod "7640641e-c713-4a9f-ab1e-0ee19b6f02aa" (UID: "7640641e-c713-4a9f-ab1e-0ee19b6f02aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:39.463505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.463417 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-kube-api-access-hcrwk" pod "7640641e-c713-4a9f-ab1e-0ee19b6f02aa" (UID: "7640641e-c713-4a9f-ab1e-0ee19b6f02aa"). InnerVolumeSpecName "kube-api-access-hcrwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:39.508448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.508410 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32778100-8a2b-46c9-b507-2273c72e10af" path="/var/lib/kubelet/pods/32778100-8a2b-46c9-b507-2273c72e10af/volumes" Nov 02 20:33:39.509018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.508997 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4d7297-e99a-4906-abd0-af7566f01302" path="/var/lib/kubelet/pods/ff4d7297-e99a-4906-abd0-af7566f01302/volumes" Nov 02 20:33:39.562750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.562707 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:39.562750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:39.562741 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcrwk\" (UniqueName: \"kubernetes.io/projected/7640641e-c713-4a9f-ab1e-0ee19b6f02aa-kube-api-access-hcrwk\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:40.203059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:40.203029 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" Nov 02 20:33:40.203480 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:40.203028 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm" event={"ID":"7640641e-c713-4a9f-ab1e-0ee19b6f02aa","Type":"ContainerDied","Data":"3ae573a842ebb373dcab409746c178b77a3cbb614d72f8da8adac0ce512757c6"} Nov 02 20:33:40.203480 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:40.203156 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ae573a842ebb373dcab409746c178b77a3cbb614d72f8da8adac0ce512757c6" Nov 02 20:33:44.652766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:44.652734 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7"] Nov 02 20:33:44.653200 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:44.652938 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" containerID="cri-o://aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850" gracePeriod=30 Nov 02 20:33:48.064298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.054106 5399 patch_prober.go:29] interesting pod/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:33:48.064298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.054337 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:33:48.111392 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.111341 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" containerID="cri-o://9899cd035cc91d4583726b8c592c97376af81389da191ad0cdddc0b2b4e36b44" gracePeriod=45 Nov 02 20:33:48.121518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.121345 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" containerID="cri-o://484cc73fd665feb190b05e4bc410c014306a77a1c56cd1b9071c5262e4cbaaa8" gracePeriod=45 Nov 02 20:33:48.123531 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.123501 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused" Nov 02 20:33:48.238678 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:33:48.238527 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:33:48.281486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.281455 5399 generic.go:356] "Generic (PLEG): container finished" podID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerID="484cc73fd665feb190b05e4bc410c014306a77a1c56cd1b9071c5262e4cbaaa8" exitCode=0 Nov 02 20:33:48.281703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.281594 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" event={"ID":"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59","Type":"ContainerDied","Data":"484cc73fd665feb190b05e4bc410c014306a77a1c56cd1b9071c5262e4cbaaa8"} Nov 02 20:33:48.285685 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.285504 5399 generic.go:356] "Generic (PLEG): container finished" podID="201c4192-e93b-428e-9695-c3c6d9585d83" containerID="9899cd035cc91d4583726b8c592c97376af81389da191ad0cdddc0b2b4e36b44" exitCode=0 Nov 02 20:33:48.285685 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.285627 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" event={"ID":"201c4192-e93b-428e-9695-c3c6d9585d83","Type":"ContainerDied","Data":"9899cd035cc91d4583726b8c592c97376af81389da191ad0cdddc0b2b4e36b44"} Nov 02 20:33:48.402034 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.402010 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:33:48.413772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.413743 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-home\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-home\") pod \"201c4192-e93b-428e-9695-c3c6d9585d83\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " Nov 02 20:33:48.413909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.413836 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201c4192-e93b-428e-9695-c3c6d9585d83-kube-api-access-gqk5x\" (UniqueName: \"kubernetes.io/projected/201c4192-e93b-428e-9695-c3c6d9585d83-kube-api-access-gqk5x\") pod \"201c4192-e93b-428e-9695-c3c6d9585d83\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " Nov 02 20:33:48.413909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.413857 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/201c4192-e93b-428e-9695-c3c6d9585d83-tls-certs\" (UniqueName: \"kubernetes.io/secret/201c4192-e93b-428e-9695-c3c6d9585d83-tls-certs\") pod \"201c4192-e93b-428e-9695-c3c6d9585d83\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " Nov 02 20:33:48.413909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.413907 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-model-cache\") pod \"201c4192-e93b-428e-9695-c3c6d9585d83\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " Nov 02 20:33:48.414087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.414016 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-tmp-dir\") pod \"201c4192-e93b-428e-9695-c3c6d9585d83\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " Nov 02 20:33:48.414087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.414060 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-dshm\") pod \"201c4192-e93b-428e-9695-c3c6d9585d83\" (UID: \"201c4192-e93b-428e-9695-c3c6d9585d83\") " Nov 02 20:33:48.414202 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.414020 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-home" pod "201c4192-e93b-428e-9695-c3c6d9585d83" (UID: "201c4192-e93b-428e-9695-c3c6d9585d83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.414202 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.414164 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-model-cache" pod "201c4192-e93b-428e-9695-c3c6d9585d83" (UID: "201c4192-e93b-428e-9695-c3c6d9585d83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.414395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.414258 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-tmp-dir" pod "201c4192-e93b-428e-9695-c3c6d9585d83" (UID: "201c4192-e93b-428e-9695-c3c6d9585d83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.415035 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.415006 5399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-home\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.415174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.415039 5399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-model-cache\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.415174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.415056 5399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-tmp-dir\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.416608 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.416576 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201c4192-e93b-428e-9695-c3c6d9585d83-tls-certs" pod "201c4192-e93b-428e-9695-c3c6d9585d83" (UID: "201c4192-e93b-428e-9695-c3c6d9585d83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:48.416724 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.416674 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-dshm" pod "201c4192-e93b-428e-9695-c3c6d9585d83" (UID: "201c4192-e93b-428e-9695-c3c6d9585d83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.416790 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.416776 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201c4192-e93b-428e-9695-c3c6d9585d83-kube-api-access-gqk5x" pod "201c4192-e93b-428e-9695-c3c6d9585d83" (UID: "201c4192-e93b-428e-9695-c3c6d9585d83"). InnerVolumeSpecName "kube-api-access-gqk5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:48.430575 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.430551 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:33:48.515701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.515660 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-kube-api-access-5sdbz\" (UniqueName: \"kubernetes.io/projected/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-kube-api-access-5sdbz\") pod \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " Nov 02 20:33:48.515868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.515725 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-model-cache\") pod \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " Nov 02 20:33:48.515868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.515761 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-dshm\") pod \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " Nov 02 20:33:48.515868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.515830 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tmp-dir\") pod \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " Nov 02 20:33:48.516087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.516068 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-home\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-home\") pod \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " Nov 02 20:33:48.516198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.516183 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tls-certs\") pod \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\" (UID: \"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59\") " Nov 02 20:33:48.516964 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.516935 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqk5x\" (UniqueName: \"kubernetes.io/projected/201c4192-e93b-428e-9695-c3c6d9585d83-kube-api-access-gqk5x\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.517096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.516968 5399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/201c4192-e93b-428e-9695-c3c6d9585d83-tls-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.517096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.516982 5399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/201c4192-e93b-428e-9695-c3c6d9585d83-dshm\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.517096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.517068 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tmp-dir" pod "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" (UID: "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.517356 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.517332 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-home" pod "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" (UID: "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.517732 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.517710 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-model-cache" pod "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" (UID: "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.518932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.518906 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-dshm" pod "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" (UID: "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:48.519655 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.519625 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-kube-api-access-5sdbz" pod "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" (UID: "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59"). InnerVolumeSpecName "kube-api-access-5sdbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:48.519655 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.519647 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tls-certs" pod "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" (UID: "7a27fd9a-920b-4562-8c2c-57c9cf8ffe59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:48.618349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.618221 5399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-home\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.618349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.618259 5399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tls-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.618349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.618294 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sdbz\" (UniqueName: \"kubernetes.io/projected/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-kube-api-access-5sdbz\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.618349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.618312 5399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-model-cache\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.618349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.618334 5399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-dshm\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:48.618349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:48.618349 5399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59-tmp-dir\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:49.164351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.164317 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg"] Nov 02 20:33:49.190304 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.186657 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-62d287-sv2qg"] Nov 02 20:33:49.298813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.298776 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" event={"ID":"7a27fd9a-920b-4562-8c2c-57c9cf8ffe59","Type":"ContainerDied","Data":"2f4a5b175f9ac3bd0b2f13907bebc03bc038c0be8045721b84c45a9ae0d5647c"} Nov 02 20:33:49.299008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.298838 5399 scope.go:122] "RemoveContainer" containerID="484cc73fd665feb190b05e4bc410c014306a77a1c56cd1b9071c5262e4cbaaa8" Nov 02 20:33:49.299008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.298794 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f" Nov 02 20:33:49.301270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.301238 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" event={"ID":"201c4192-e93b-428e-9695-c3c6d9585d83","Type":"ContainerDied","Data":"d625f676c311327ab7d2d6a4fc4fcaba3b8f4ec3428ae0a6d650024f02c9c1f4"} Nov 02 20:33:49.301488 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.301471 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb" Nov 02 20:33:49.333877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.333854 5399 scope.go:122] "RemoveContainer" containerID="9899cd035cc91d4583726b8c592c97376af81389da191ad0cdddc0b2b4e36b44" Nov 02 20:33:49.453427 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.453338 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f"] Nov 02 20:33:49.457891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.457861 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/premium-w1-accdf5-kserve-85488bb77f-6nc7f"] Nov 02 20:33:49.509637 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.509609 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" path="/var/lib/kubelet/pods/7a27fd9a-920b-4562-8c2c-57c9cf8ffe59/volumes" Nov 02 20:33:49.510058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.510045 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c71726f-17b8-442c-96d3-2d78f3f22ee7" path="/var/lib/kubelet/pods/7c71726f-17b8-442c-96d3-2d78f3f22ee7/volumes" Nov 02 20:33:49.559320 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.559256 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb"] Nov 02 20:33:49.562301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.562260 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-accdf5/distinct-w1-accdf5-kserve-7d948c9d94-7qktb"] Nov 02 20:33:49.930825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:49.930801 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:33:50.045515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.045430 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-certs\") pod \"8134e54f-4985-438d-9da1-010c47005ee9\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " Nov 02 20:33:50.045667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.045519 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-kube-api-access-v8spr\" (UniqueName: \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-kube-api-access-v8spr\") pod \"8134e54f-4985-438d-9da1-010c47005ee9\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " Nov 02 20:33:50.045667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.045607 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8134e54f-4985-438d-9da1-010c47005ee9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8134e54f-4985-438d-9da1-010c47005ee9-istio-podinfo\") pod \"8134e54f-4985-438d-9da1-010c47005ee9\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " Nov 02 20:33:50.045667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.045655 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-credential-socket\") pod \"8134e54f-4985-438d-9da1-010c47005ee9\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " Nov 02 20:33:50.045919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.045694 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-istio-token\" (UniqueName: \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-istio-token\") pod \"8134e54f-4985-438d-9da1-010c47005ee9\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " Nov 02 20:33:50.045919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.045877 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8134e54f-4985-438d-9da1-010c47005ee9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8134e54f-4985-438d-9da1-010c47005ee9-istiod-ca-cert\") pod \"8134e54f-4985-438d-9da1-010c47005ee9\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " Nov 02 20:33:50.045919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.045914 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-socket\") pod \"8134e54f-4985-438d-9da1-010c47005ee9\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " Nov 02 20:33:50.046091 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.046055 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-envoy\") pod \"8134e54f-4985-438d-9da1-010c47005ee9\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " Nov 02 20:33:50.046091 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.046085 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-data\") pod \"8134e54f-4985-438d-9da1-010c47005ee9\" (UID: \"8134e54f-4985-438d-9da1-010c47005ee9\") " Nov 02 20:33:50.047159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.047131 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-certs" pod "8134e54f-4985-438d-9da1-010c47005ee9" (UID: "8134e54f-4985-438d-9da1-010c47005ee9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:50.047360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.047335 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-data" pod "8134e54f-4985-438d-9da1-010c47005ee9" (UID: "8134e54f-4985-438d-9da1-010c47005ee9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:50.047569 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.047546 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-socket" pod "8134e54f-4985-438d-9da1-010c47005ee9" (UID: "8134e54f-4985-438d-9da1-010c47005ee9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:50.047710 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.047567 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8134e54f-4985-438d-9da1-010c47005ee9-istiod-ca-cert" pod "8134e54f-4985-438d-9da1-010c47005ee9" (UID: "8134e54f-4985-438d-9da1-010c47005ee9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:50.051449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.051409 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8134e54f-4985-438d-9da1-010c47005ee9-istio-podinfo" pod "8134e54f-4985-438d-9da1-010c47005ee9" (UID: "8134e54f-4985-438d-9da1-010c47005ee9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:33:50.051695 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.051660 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-envoy" pod "8134e54f-4985-438d-9da1-010c47005ee9" (UID: "8134e54f-4985-438d-9da1-010c47005ee9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:50.051772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.051711 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-istio-token" pod "8134e54f-4985-438d-9da1-010c47005ee9" (UID: "8134e54f-4985-438d-9da1-010c47005ee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:50.053193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.053163 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-credential-socket" pod "8134e54f-4985-438d-9da1-010c47005ee9" (UID: "8134e54f-4985-438d-9da1-010c47005ee9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:50.053545 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.053501 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-kube-api-access-v8spr" pod "8134e54f-4985-438d-9da1-010c47005ee9" (UID: "8134e54f-4985-438d-9da1-010c47005ee9"). InnerVolumeSpecName "kube-api-access-v8spr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:50.147906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.147873 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.147906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.147904 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8spr\" (UniqueName: \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-kube-api-access-v8spr\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.147906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.147913 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8134e54f-4985-438d-9da1-010c47005ee9-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.148136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.147923 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.148136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.147931 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8134e54f-4985-438d-9da1-010c47005ee9-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.148136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.147939 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8134e54f-4985-438d-9da1-010c47005ee9-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.148136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.147948 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.148136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.147956 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.148136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.147964 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8134e54f-4985-438d-9da1-010c47005ee9-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:50.312010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.311976 5399 generic.go:356] "Generic (PLEG): container finished" podID="8134e54f-4985-438d-9da1-010c47005ee9" containerID="aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850" exitCode=0 Nov 02 20:33:50.312511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.312023 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" Nov 02 20:33:50.312511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.312059 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" event={"ID":"8134e54f-4985-438d-9da1-010c47005ee9","Type":"ContainerDied","Data":"aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850"} Nov 02 20:33:50.312511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.312110 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7" event={"ID":"8134e54f-4985-438d-9da1-010c47005ee9","Type":"ContainerDied","Data":"c46bed2f78255c1c9db09b4387fb6ba80ec0dc4104e79f59ad51de84a1036ec2"} Nov 02 20:33:50.312511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.312137 5399 scope.go:122] "RemoveContainer" containerID="aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850" Nov 02 20:33:50.323889 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.323871 5399 scope.go:122] "RemoveContainer" containerID="aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850" Nov 02 20:33:50.324135 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:33:50.324117 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850\": container with ID starting with aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850 not found: ID does not exist" containerID="aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850" Nov 02 20:33:50.324179 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.324143 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850"} err="failed to get container status \"aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850\": rpc error: code = NotFound desc = could not find container \"aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850\": container with ID starting with aa84456c081866bdbbb07ca91f270f2872de29e1e6b2cc8d86a16ac10622f850 not found: ID does not exist" Nov 02 20:33:50.468782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.468721 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7"] Nov 02 20:33:50.515055 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:50.515023 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9fcedd-openshift-default-7f55689787-nr7l7"] Nov 02 20:33:51.508595 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:51.508562 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" path="/var/lib/kubelet/pods/201c4192-e93b-428e-9695-c3c6d9585d83/volumes" Nov 02 20:33:51.509069 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:51.509054 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8134e54f-4985-438d-9da1-010c47005ee9" path="/var/lib/kubelet/pods/8134e54f-4985-438d-9da1-010c47005ee9/volumes" Nov 02 20:33:58.205085 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205050 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf"] Nov 02 20:33:58.205613 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205600 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7640641e-c713-4a9f-ab1e-0ee19b6f02aa" containerName="revoke-keys" Nov 02 20:33:58.205654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205615 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7640641e-c713-4a9f-ab1e-0ee19b6f02aa" containerName="revoke-keys" Nov 02 20:33:58.205654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205630 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" Nov 02 20:33:58.205654 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205635 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" Nov 02 20:33:58.205760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205664 5399 cpu_manager.go:414] "containerMap: removing container" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" Nov 02 20:33:58.205760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205669 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" Nov 02 20:33:58.205760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205677 5399 cpu_manager.go:414] "containerMap: removing container" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" Nov 02 20:33:58.205760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205682 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" Nov 02 20:33:58.205760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205700 5399 cpu_manager.go:414] "containerMap: removing container" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" Nov 02 20:33:58.205760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205705 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" Nov 02 20:33:58.205975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205795 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" Nov 02 20:33:58.205975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205803 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="8134e54f-4985-438d-9da1-010c47005ee9" containerName="istio-proxy" Nov 02 20:33:58.205975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205810 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="201c4192-e93b-428e-9695-c3c6d9585d83" containerName="main" Nov 02 20:33:58.205975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205815 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" Nov 02 20:33:58.205975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205822 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7640641e-c713-4a9f-ab1e-0ee19b6f02aa" containerName="revoke-keys" Nov 02 20:33:58.205975 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.205828 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a27fd9a-920b-4562-8c2c-57c9cf8ffe59" containerName="main" Nov 02 20:33:58.210783 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.210764 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.261356 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.261323 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf"] Nov 02 20:33:58.303580 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.303538 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:33:58.303580 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.303562 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:33:58.361042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.361007 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.361195 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.361067 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjbv\" (UniqueName: \"kubernetes.io/projected/d85848fb-eb58-42ad-8598-57b13cdadd84-kube-api-access-9wjbv\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.361257 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.361208 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.462095 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.462009 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjbv\" (UniqueName: \"kubernetes.io/projected/d85848fb-eb58-42ad-8598-57b13cdadd84-kube-api-access-9wjbv\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.462257 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.462165 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.462490 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.462375 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.462994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.462970 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.464450 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.464430 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.514679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.514647 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjbv\" (UniqueName: \"kubernetes.io/projected/d85848fb-eb58-42ad-8598-57b13cdadd84-kube-api-access-9wjbv\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.524170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.524139 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:33:58.893982 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:58.893953 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf"] Nov 02 20:33:58.894869 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:33:58.894839 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85848fb_eb58_42ad_8598_57b13cdadd84.slice/crio-da3121db5656b91edaed02036c1be8d4208152beb8a087d7ad13706e73c06c6f WatchSource:0}: Error finding container da3121db5656b91edaed02036c1be8d4208152beb8a087d7ad13706e73c06c6f: Status 404 returned error can't find the container with id da3121db5656b91edaed02036c1be8d4208152beb8a087d7ad13706e73c06c6f Nov 02 20:33:59.391120 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:59.391079 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" event={"ID":"d85848fb-eb58-42ad-8598-57b13cdadd84","Type":"ContainerStarted","Data":"de7c2069e200cabef9de8da723dafa9fd34e4e2d7312e012b1c2e0a08d9d2f90"} Nov 02 20:33:59.391610 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:59.391140 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" event={"ID":"d85848fb-eb58-42ad-8598-57b13cdadd84","Type":"ContainerStarted","Data":"da3121db5656b91edaed02036c1be8d4208152beb8a087d7ad13706e73c06c6f"} Nov 02 20:33:59.499518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:33:59.499423 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" podStartSLOduration=1.499408632 podStartE2EDuration="1.499408632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:59.496242468 +0000 UTC m=+2482.106894560" watchObservedRunningTime="2026-11-02 20:33:59.499408632 +0000 UTC m=+2482.110060736" Nov 02 20:34:00.400069 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:00.400031 5399 generic.go:356] "Generic (PLEG): container finished" podID="d85848fb-eb58-42ad-8598-57b13cdadd84" containerID="de7c2069e200cabef9de8da723dafa9fd34e4e2d7312e012b1c2e0a08d9d2f90" exitCode=0 Nov 02 20:34:00.400515 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:00.400120 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" event={"ID":"d85848fb-eb58-42ad-8598-57b13cdadd84","Type":"ContainerDied","Data":"de7c2069e200cabef9de8da723dafa9fd34e4e2d7312e012b1c2e0a08d9d2f90"} Nov 02 20:34:01.558562 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:01.558536 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:34:01.705046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:01.704956 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-openshift-ca\") pod \"d85848fb-eb58-42ad-8598-57b13cdadd84\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " Nov 02 20:34:01.705286 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:01.705168 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-xks-ca\") pod \"d85848fb-eb58-42ad-8598-57b13cdadd84\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " Nov 02 20:34:01.705286 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:01.705218 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85848fb-eb58-42ad-8598-57b13cdadd84-kube-api-access-9wjbv\" (UniqueName: \"kubernetes.io/projected/d85848fb-eb58-42ad-8598-57b13cdadd84-kube-api-access-9wjbv\") pod \"d85848fb-eb58-42ad-8598-57b13cdadd84\" (UID: \"d85848fb-eb58-42ad-8598-57b13cdadd84\") " Nov 02 20:34:01.705482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:01.705456 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-openshift-ca" pod "d85848fb-eb58-42ad-8598-57b13cdadd84" (UID: "d85848fb-eb58-42ad-8598-57b13cdadd84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:34:01.705748 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:01.705729 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:01.707619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:01.707590 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-xks-ca" pod "d85848fb-eb58-42ad-8598-57b13cdadd84" (UID: "d85848fb-eb58-42ad-8598-57b13cdadd84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:34:01.707736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:01.707612 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85848fb-eb58-42ad-8598-57b13cdadd84-kube-api-access-9wjbv" pod "d85848fb-eb58-42ad-8598-57b13cdadd84" (UID: "d85848fb-eb58-42ad-8598-57b13cdadd84"). InnerVolumeSpecName "kube-api-access-9wjbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:01.806649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:01.806611 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85848fb-eb58-42ad-8598-57b13cdadd84-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:01.806649 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:01.806642 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wjbv\" (UniqueName: \"kubernetes.io/projected/d85848fb-eb58-42ad-8598-57b13cdadd84-kube-api-access-9wjbv\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:02.416998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:02.416966 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" Nov 02 20:34:02.416998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:02.416986 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf" event={"ID":"d85848fb-eb58-42ad-8598-57b13cdadd84","Type":"ContainerDied","Data":"da3121db5656b91edaed02036c1be8d4208152beb8a087d7ad13706e73c06c6f"} Nov 02 20:34:02.417228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:02.417043 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da3121db5656b91edaed02036c1be8d4208152beb8a087d7ad13706e73c06c6f" Nov 02 20:34:09.954927 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:09.954884 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:34:18.237004 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:34:18.236954 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:34:40.342528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:40.342489 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw"] Nov 02 20:34:40.342956 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:40.342722 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" containerID="cri-o://345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506" gracePeriod=30 Nov 02 20:34:40.388536 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:40.388499 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4"] Nov 02 20:34:40.388783 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:40.388758 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" containerID="cri-o://f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf" gracePeriod=30 Nov 02 20:34:44.301923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.301885 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t"] Nov 02 20:34:44.302733 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.302709 5399 cpu_manager.go:414] "containerMap: removing container" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" Nov 02 20:34:44.302835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.302734 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32778100-8a2b-46c9-b507-2273c72e10af" containerName="istio-proxy" Nov 02 20:34:44.302835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.302758 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d85848fb-eb58-42ad-8598-57b13cdadd84" containerName="revoke-keys" Nov 02 20:34:44.302835 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.302767 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85848fb-eb58-42ad-8598-57b13cdadd84" containerName="revoke-keys" Nov 02 20:34:44.302995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.302915 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85848fb-eb58-42ad-8598-57b13cdadd84" containerName="revoke-keys" Nov 02 20:34:44.307763 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.307741 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:44.353758 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.353719 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d"] Nov 02 20:34:44.356622 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.356588 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:34:44.358202 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.358177 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.358336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.358252 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t"] Nov 02 20:34:44.396565 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.396531 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:34:44.420147 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.420115 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d"] Nov 02 20:34:44.468554 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.468522 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fed7f16-msz6d\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.468754 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.468582 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86dr\" (UniqueName: \"kubernetes.io/projected/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-kube-api-access-m86dr\") pod \"maas-api-revoke-sub-2fed7f16-msz6d\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.468754 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.468712 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fa60f03-zwt5t\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:44.468910 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.468887 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fed7f16-msz6d\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.468955 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.468934 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5q5q\" (UniqueName: \"kubernetes.io/projected/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-kube-api-access-z5q5q\") pod \"maas-api-revoke-sub-6fa60f03-zwt5t\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:44.469030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.468961 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fa60f03-zwt5t\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:44.570256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.570217 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fed7f16-msz6d\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.570483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.570344 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86dr\" (UniqueName: \"kubernetes.io/projected/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-kube-api-access-m86dr\") pod \"maas-api-revoke-sub-2fed7f16-msz6d\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.570483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.570406 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fa60f03-zwt5t\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:44.570615 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.570547 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fed7f16-msz6d\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.570615 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.570598 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5q5q\" (UniqueName: \"kubernetes.io/projected/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-kube-api-access-z5q5q\") pod \"maas-api-revoke-sub-6fa60f03-zwt5t\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:44.570723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.570633 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fa60f03-zwt5t\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:44.571372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.571340 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fed7f16-msz6d\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.571506 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.571410 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fa60f03-zwt5t\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:44.572718 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.572698 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fa60f03-zwt5t\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:44.572797 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.572765 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fed7f16-msz6d\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.623030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.622993 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86dr\" (UniqueName: \"kubernetes.io/projected/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-kube-api-access-m86dr\") pod \"maas-api-revoke-sub-2fed7f16-msz6d\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.623197 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.623172 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5q5q\" (UniqueName: \"kubernetes.io/projected/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-kube-api-access-z5q5q\") pod \"maas-api-revoke-sub-6fa60f03-zwt5t\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:44.673395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.673365 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:44.921517 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:44.921429 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:45.039441 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.039418 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d"] Nov 02 20:34:45.041742 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:34:45.041710 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82e5ffc_e0b0_4507_9507_ff96f1e3fd7e.slice/crio-91e5f0a54c902d3305392e5fed3fa9939e636c8489aee09c768a60cccd82e28d WatchSource:0}: Error finding container 91e5f0a54c902d3305392e5fed3fa9939e636c8489aee09c768a60cccd82e28d: Status 404 returned error can't find the container with id 91e5f0a54c902d3305392e5fed3fa9939e636c8489aee09c768a60cccd82e28d Nov 02 20:34:45.291247 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.288976 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t"] Nov 02 20:34:45.478974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.478939 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-qqdvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:34:45.479459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.479005 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 20:34:45.583786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.583671 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-jg6q4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 02 20:34:45.583786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.583733 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 02 20:34:45.674315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.674268 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:34:45.700461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.700434 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:34:45.744096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.744054 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" event={"ID":"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e","Type":"ContainerStarted","Data":"9e99f0dcfab25ed66267544bec2690dc5be7678fa0ad0d981332ecc45a8aede4"} Nov 02 20:34:45.744295 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.744129 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" event={"ID":"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e","Type":"ContainerStarted","Data":"91e5f0a54c902d3305392e5fed3fa9939e636c8489aee09c768a60cccd82e28d"} Nov 02 20:34:45.745731 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.745704 5399 generic.go:356] "Generic (PLEG): container finished" podID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerID="f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf" exitCode=0 Nov 02 20:34:45.745839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.745750 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" Nov 02 20:34:45.745839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.745786 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" event={"ID":"39ed5d06-4f0e-4264-9219-bf19ab89dbff","Type":"ContainerDied","Data":"f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf"} Nov 02 20:34:45.745839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.745827 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4" event={"ID":"39ed5d06-4f0e-4264-9219-bf19ab89dbff","Type":"ContainerDied","Data":"fc81d1d7154b7b00906f56b50fbab70f6bfc1a7c8179737862820d13074d9f9d"} Nov 02 20:34:45.745995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.745850 5399 scope.go:122] "RemoveContainer" containerID="f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf" Nov 02 20:34:45.747352 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.747327 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" event={"ID":"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42","Type":"ContainerStarted","Data":"064488a7c6fbb55c312777b4929ebd6042772a6c7cfea33c3ccc001d20331ac4"} Nov 02 20:34:45.747472 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.747377 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" event={"ID":"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42","Type":"ContainerStarted","Data":"4d30753a534e4068b3c4f4ebe5284d3f1a229fd2a8b8587406b014b9e35e213a"} Nov 02 20:34:45.748690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.748671 5399 generic.go:356] "Generic (PLEG): container finished" podID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerID="345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506" exitCode=0 Nov 02 20:34:45.748785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.748708 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" Nov 02 20:34:45.748785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.748711 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" event={"ID":"aa2137fc-c108-477f-b6ee-8d36b7e63189","Type":"ContainerDied","Data":"345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506"} Nov 02 20:34:45.748785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.748763 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw" event={"ID":"aa2137fc-c108-477f-b6ee-8d36b7e63189","Type":"ContainerDied","Data":"95e35395b6e98916c67a3701eec2798d3588202079076d8274d9118d21fe1bf7"} Nov 02 20:34:45.756222 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.756204 5399 scope.go:122] "RemoveContainer" containerID="f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf" Nov 02 20:34:45.756523 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:34:45.756505 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf\": container with ID starting with f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf not found: ID does not exist" containerID="f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf" Nov 02 20:34:45.756604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.756536 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf"} err="failed to get container status \"f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf\": rpc error: code = NotFound desc = could not find container \"f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf\": container with ID starting with f4143f089f868c7436c02a2a0e6ce5f0df1dd422bb56e17a8a2fa0bdb859dfdf not found: ID does not exist" Nov 02 20:34:45.756604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.756560 5399 scope.go:122] "RemoveContainer" containerID="345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506" Nov 02 20:34:45.765917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.765892 5399 scope.go:122] "RemoveContainer" containerID="345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506" Nov 02 20:34:45.766163 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:34:45.766145 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506\": container with ID starting with 345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506 not found: ID does not exist" containerID="345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506" Nov 02 20:34:45.766212 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.766174 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506"} err="failed to get container status \"345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506\": rpc error: code = NotFound desc = could not find container \"345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506\": container with ID starting with 345c10e02057c243357f44488a55432834aca0d88984bae0f9027a33ff94d506 not found: ID does not exist" Nov 02 20:34:45.786723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.786695 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-envoy\") pod \"aa2137fc-c108-477f-b6ee-8d36b7e63189\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " Nov 02 20:34:45.786868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.786779 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-credential-socket\") pod \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " Nov 02 20:34:45.786868 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.786829 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-envoy\") pod \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " Nov 02 20:34:45.786989 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.786875 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-certs\") pod \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " Nov 02 20:34:45.786989 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.786970 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-kube-api-access-mm2f8\" (UniqueName: \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-kube-api-access-mm2f8\") pod \"aa2137fc-c108-477f-b6ee-8d36b7e63189\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " Nov 02 20:34:45.787086 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787016 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-kube-api-access-bhf8n\" (UniqueName: \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-kube-api-access-bhf8n\") pod \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " Nov 02 20:34:45.787086 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787052 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-credential-socket" pod "39ed5d06-4f0e-4264-9219-bf19ab89dbff" (UID: "39ed5d06-4f0e-4264-9219-bf19ab89dbff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:45.787344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787314 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-certs" pod "39ed5d06-4f0e-4264-9219-bf19ab89dbff" (UID: "39ed5d06-4f0e-4264-9219-bf19ab89dbff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:45.787468 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787419 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-socket\") pod \"aa2137fc-c108-477f-b6ee-8d36b7e63189\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " Nov 02 20:34:45.787468 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787457 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-data\") pod \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " Nov 02 20:34:45.787592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787499 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-podinfo\") pod \"aa2137fc-c108-477f-b6ee-8d36b7e63189\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " Nov 02 20:34:45.787592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787529 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-credential-socket\") pod \"aa2137fc-c108-477f-b6ee-8d36b7e63189\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " Nov 02 20:34:45.787592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787584 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-token\" (UniqueName: \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-token\") pod \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " Nov 02 20:34:45.787813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787624 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-socket\") pod \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " Nov 02 20:34:45.787813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787680 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-token\" (UniqueName: \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-token\") pod \"aa2137fc-c108-477f-b6ee-8d36b7e63189\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " Nov 02 20:34:45.787813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787709 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-certs\") pod \"aa2137fc-c108-477f-b6ee-8d36b7e63189\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " Nov 02 20:34:45.787813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787752 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2137fc-c108-477f-b6ee-8d36b7e63189-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2137fc-c108-477f-b6ee-8d36b7e63189-istiod-ca-cert\") pod \"aa2137fc-c108-477f-b6ee-8d36b7e63189\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " Nov 02 20:34:45.787813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787779 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-podinfo\") pod \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " Nov 02 20:34:45.788060 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787844 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istiod-ca-cert\") pod \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\" (UID: \"39ed5d06-4f0e-4264-9219-bf19ab89dbff\") " Nov 02 20:34:45.788060 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.787890 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-data\") pod \"aa2137fc-c108-477f-b6ee-8d36b7e63189\" (UID: \"aa2137fc-c108-477f-b6ee-8d36b7e63189\") " Nov 02 20:34:45.788442 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.788425 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.788709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.788446 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.788884 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.788860 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-data" pod "aa2137fc-c108-477f-b6ee-8d36b7e63189" (UID: "aa2137fc-c108-477f-b6ee-8d36b7e63189"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:45.789070 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.789051 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-socket" pod "aa2137fc-c108-477f-b6ee-8d36b7e63189" (UID: "aa2137fc-c108-477f-b6ee-8d36b7e63189"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:45.789320 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.789298 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-data" pod "39ed5d06-4f0e-4264-9219-bf19ab89dbff" (UID: "39ed5d06-4f0e-4264-9219-bf19ab89dbff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:45.789593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.789567 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-kube-api-access-bhf8n" pod "39ed5d06-4f0e-4264-9219-bf19ab89dbff" (UID: "39ed5d06-4f0e-4264-9219-bf19ab89dbff"). InnerVolumeSpecName "kube-api-access-bhf8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:45.789699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.789653 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-kube-api-access-mm2f8" pod "aa2137fc-c108-477f-b6ee-8d36b7e63189" (UID: "aa2137fc-c108-477f-b6ee-8d36b7e63189"). InnerVolumeSpecName "kube-api-access-mm2f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:45.789852 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.789828 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-envoy" pod "aa2137fc-c108-477f-b6ee-8d36b7e63189" (UID: "aa2137fc-c108-477f-b6ee-8d36b7e63189"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:45.789915 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.789892 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2137fc-c108-477f-b6ee-8d36b7e63189-istiod-ca-cert" pod "aa2137fc-c108-477f-b6ee-8d36b7e63189" (UID: "aa2137fc-c108-477f-b6ee-8d36b7e63189"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:34:45.789999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.789978 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-envoy" pod "39ed5d06-4f0e-4264-9219-bf19ab89dbff" (UID: "39ed5d06-4f0e-4264-9219-bf19ab89dbff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:45.790228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.790206 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-certs" pod "aa2137fc-c108-477f-b6ee-8d36b7e63189" (UID: "aa2137fc-c108-477f-b6ee-8d36b7e63189"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:45.790345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.790221 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-credential-socket" pod "aa2137fc-c108-477f-b6ee-8d36b7e63189" (UID: "aa2137fc-c108-477f-b6ee-8d36b7e63189"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:45.790448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.790426 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-socket" pod "39ed5d06-4f0e-4264-9219-bf19ab89dbff" (UID: "39ed5d06-4f0e-4264-9219-bf19ab89dbff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:34:45.790574 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.790551 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istiod-ca-cert" pod "39ed5d06-4f0e-4264-9219-bf19ab89dbff" (UID: "39ed5d06-4f0e-4264-9219-bf19ab89dbff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:34:45.791549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.791514 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-podinfo" pod "39ed5d06-4f0e-4264-9219-bf19ab89dbff" (UID: "39ed5d06-4f0e-4264-9219-bf19ab89dbff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:34:45.791863 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.791721 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-podinfo" pod "aa2137fc-c108-477f-b6ee-8d36b7e63189" (UID: "aa2137fc-c108-477f-b6ee-8d36b7e63189"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:34:45.792025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.791996 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-token" pod "39ed5d06-4f0e-4264-9219-bf19ab89dbff" (UID: "39ed5d06-4f0e-4264-9219-bf19ab89dbff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:45.792322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.792300 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-token" pod "aa2137fc-c108-477f-b6ee-8d36b7e63189" (UID: "aa2137fc-c108-477f-b6ee-8d36b7e63189"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:45.889948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.889917 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm2f8\" (UniqueName: \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-kube-api-access-mm2f8\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890050 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhf8n\" (UniqueName: \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-kube-api-access-bhf8n\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890074 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890090 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890103 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890116 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890129 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890142 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890156 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890169 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890184 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2137fc-c108-477f-b6ee-8d36b7e63189-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890198 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890212 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890226 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890240 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2137fc-c108-477f-b6ee-8d36b7e63189-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:45.890428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:45.890255 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ed5d06-4f0e-4264-9219-bf19ab89dbff-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:46.000657 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:46.000450 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" podStartSLOduration=2.000430947 podStartE2EDuration="2.000430947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:45.997178631 +0000 UTC m=+2528.607830724" watchObservedRunningTime="2026-11-02 20:34:46.000430947 +0000 UTC m=+2528.611083039" Nov 02 20:34:46.153833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:46.153770 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" podStartSLOduration=2.153755616 podStartE2EDuration="2.153755616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:46.150324586 +0000 UTC m=+2528.760976681" watchObservedRunningTime="2026-11-02 20:34:46.153755616 +0000 UTC m=+2528.764407718" Nov 02 20:34:46.255903 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:46.255823 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4"] Nov 02 20:34:46.258930 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:46.258905 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-jg6q4"] Nov 02 20:34:46.361603 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:46.361570 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw"] Nov 02 20:34:46.365119 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:46.365093 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-qqdvw"] Nov 02 20:34:46.759111 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:46.759079 5399 generic.go:356] "Generic (PLEG): container finished" podID="c5b1bec4-3a9e-43dd-b96f-f36af89dbc42" containerID="064488a7c6fbb55c312777b4929ebd6042772a6c7cfea33c3ccc001d20331ac4" exitCode=0 Nov 02 20:34:46.759572 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:46.759159 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" event={"ID":"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42","Type":"ContainerDied","Data":"064488a7c6fbb55c312777b4929ebd6042772a6c7cfea33c3ccc001d20331ac4"} Nov 02 20:34:46.761685 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:46.761660 5399 generic.go:356] "Generic (PLEG): container finished" podID="b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e" containerID="9e99f0dcfab25ed66267544bec2690dc5be7678fa0ad0d981332ecc45a8aede4" exitCode=0 Nov 02 20:34:46.761816 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:46.761729 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" event={"ID":"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e","Type":"ContainerDied","Data":"9e99f0dcfab25ed66267544bec2690dc5be7678fa0ad0d981332ecc45a8aede4"} Nov 02 20:34:47.508009 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:47.507972 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" path="/var/lib/kubelet/pods/39ed5d06-4f0e-4264-9219-bf19ab89dbff/volumes" Nov 02 20:34:47.508603 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:47.508580 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" path="/var/lib/kubelet/pods/aa2137fc-c108-477f-b6ee-8d36b7e63189/volumes" Nov 02 20:34:47.918717 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:47.918688 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:47.952120 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:47.952092 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:48.016291 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.016250 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-xks-ca\") pod \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " Nov 02 20:34:48.016438 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.016323 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-openshift-ca\") pod \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " Nov 02 20:34:48.016438 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.016416 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-kube-api-access-z5q5q\" (UniqueName: \"kubernetes.io/projected/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-kube-api-access-z5q5q\") pod \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\" (UID: \"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42\") " Nov 02 20:34:48.016617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.016599 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-openshift-ca" pod "c5b1bec4-3a9e-43dd-b96f-f36af89dbc42" (UID: "c5b1bec4-3a9e-43dd-b96f-f36af89dbc42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:34:48.016878 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.016866 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:48.018348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.018319 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-xks-ca" pod "c5b1bec4-3a9e-43dd-b96f-f36af89dbc42" (UID: "c5b1bec4-3a9e-43dd-b96f-f36af89dbc42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:34:48.018449 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.018427 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-kube-api-access-z5q5q" pod "c5b1bec4-3a9e-43dd-b96f-f36af89dbc42" (UID: "c5b1bec4-3a9e-43dd-b96f-f36af89dbc42"). InnerVolumeSpecName "kube-api-access-z5q5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:48.117916 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.117885 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-xks-ca\") pod \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " Nov 02 20:34:48.118108 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.117930 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-kube-api-access-m86dr\" (UniqueName: \"kubernetes.io/projected/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-kube-api-access-m86dr\") pod \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " Nov 02 20:34:48.118108 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.118011 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-openshift-ca\") pod \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\" (UID: \"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e\") " Nov 02 20:34:48.118477 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.118450 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-openshift-ca" pod "b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e" (UID: "b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:34:48.118641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.118608 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:48.118641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.118632 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5q5q\" (UniqueName: \"kubernetes.io/projected/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42-kube-api-access-z5q5q\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:48.118641 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.118642 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:48.120013 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.119994 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-xks-ca" pod "b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e" (UID: "b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:34:48.120094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.120064 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-kube-api-access-m86dr" pod "b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e" (UID: "b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e"). InnerVolumeSpecName "kube-api-access-m86dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:34:48.220428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.220399 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:48.220428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.220425 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86dr\" (UniqueName: \"kubernetes.io/projected/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e-kube-api-access-m86dr\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:34:48.238734 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:34:48.238696 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:34:48.777955 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.777923 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" Nov 02 20:34:48.778157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.777961 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t" event={"ID":"c5b1bec4-3a9e-43dd-b96f-f36af89dbc42","Type":"ContainerDied","Data":"4d30753a534e4068b3c4f4ebe5284d3f1a229fd2a8b8587406b014b9e35e213a"} Nov 02 20:34:48.778157 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.778033 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d30753a534e4068b3c4f4ebe5284d3f1a229fd2a8b8587406b014b9e35e213a" Nov 02 20:34:48.779409 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.779383 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" Nov 02 20:34:48.779549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.779429 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d" event={"ID":"b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e","Type":"ContainerDied","Data":"91e5f0a54c902d3305392e5fed3fa9939e636c8489aee09c768a60cccd82e28d"} Nov 02 20:34:48.779549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:48.779465 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91e5f0a54c902d3305392e5fed3fa9939e636c8489aee09c768a60cccd82e28d" Nov 02 20:34:55.624323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:55.624255 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm"] Nov 02 20:34:55.670941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:55.670901 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4dc8ca-grjbm"] Nov 02 20:34:57.508180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:34:57.508143 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7640641e-c713-4a9f-ab1e-0ee19b6f02aa" path="/var/lib/kubelet/pods/7640641e-c713-4a9f-ab1e-0ee19b6f02aa/volumes" Nov 02 20:35:02.914459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.914422 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd"] Nov 02 20:35:02.914979 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.914965 5399 cpu_manager.go:414] "containerMap: removing container" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" Nov 02 20:35:02.915029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.914980 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" Nov 02 20:35:02.915029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.915001 5399 cpu_manager.go:414] "containerMap: removing container" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" Nov 02 20:35:02.915029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.915006 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" Nov 02 20:35:02.915029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.915018 5399 cpu_manager.go:414] "containerMap: removing container" podUID="b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e" containerName="revoke-keys" Nov 02 20:35:02.915029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.915023 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e" containerName="revoke-keys" Nov 02 20:35:02.915177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.915032 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c5b1bec4-3a9e-43dd-b96f-f36af89dbc42" containerName="revoke-keys" Nov 02 20:35:02.915177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.915037 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b1bec4-3a9e-43dd-b96f-f36af89dbc42" containerName="revoke-keys" Nov 02 20:35:02.915177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.915127 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2137fc-c108-477f-b6ee-8d36b7e63189" containerName="istio-proxy" Nov 02 20:35:02.915177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.915137 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e" containerName="revoke-keys" Nov 02 20:35:02.915177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.915143 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b1bec4-3a9e-43dd-b96f-f36af89dbc42" containerName="revoke-keys" Nov 02 20:35:02.915177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.915157 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ed5d06-4f0e-4264-9219-bf19ab89dbff" containerName="istio-proxy" Nov 02 20:35:02.918643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.918621 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:02.972780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:02.972748 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd"] Nov 02 20:35:03.014017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.013974 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:35:03.014197 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.014082 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:35:03.025141 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.025110 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:03.025351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.025167 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:03.025351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.025257 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9g2p\" (UniqueName: \"kubernetes.io/projected/4e407044-cf41-4488-a6eb-720ea25a1e4f-kube-api-access-d9g2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:03.126504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.126463 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:03.126719 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.126533 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:03.126719 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.126567 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9g2p\" (UniqueName: \"kubernetes.io/projected/4e407044-cf41-4488-a6eb-720ea25a1e4f-kube-api-access-d9g2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:03.127138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.127112 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:03.128848 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.128827 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:03.179352 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.179251 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9g2p\" (UniqueName: \"kubernetes.io/projected/4e407044-cf41-4488-a6eb-720ea25a1e4f-kube-api-access-d9g2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:03.233046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.233009 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:03.603003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.602974 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd"] Nov 02 20:35:03.604076 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:35:03.604046 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e407044_cf41_4488_a6eb_720ea25a1e4f.slice/crio-459c5d96547765ec9ebef9dd0ab10da97ee70bd1a22fcad3752175b70de9ef9c WatchSource:0}: Error finding container 459c5d96547765ec9ebef9dd0ab10da97ee70bd1a22fcad3752175b70de9ef9c: Status 404 returned error can't find the container with id 459c5d96547765ec9ebef9dd0ab10da97ee70bd1a22fcad3752175b70de9ef9c Nov 02 20:35:03.889436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:03.889327 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" event={"ID":"4e407044-cf41-4488-a6eb-720ea25a1e4f","Type":"ContainerStarted","Data":"459c5d96547765ec9ebef9dd0ab10da97ee70bd1a22fcad3752175b70de9ef9c"} Nov 02 20:35:04.898354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:04.898270 5399 generic.go:356] "Generic (PLEG): container finished" podID="4e407044-cf41-4488-a6eb-720ea25a1e4f" containerID="2959119aabc9b8f0fb416222973b1f63727dcfeefa66358d14515e9763685715" exitCode=0 Nov 02 20:35:04.898766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:04.898329 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" event={"ID":"4e407044-cf41-4488-a6eb-720ea25a1e4f","Type":"ContainerDied","Data":"2959119aabc9b8f0fb416222973b1f63727dcfeefa66358d14515e9763685715"} Nov 02 20:35:06.067953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.067925 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:06.167970 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.167872 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-xks-ca\") pod \"4e407044-cf41-4488-a6eb-720ea25a1e4f\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " Nov 02 20:35:06.167970 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.167922 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e407044-cf41-4488-a6eb-720ea25a1e4f-kube-api-access-d9g2p\" (UniqueName: \"kubernetes.io/projected/4e407044-cf41-4488-a6eb-720ea25a1e4f-kube-api-access-d9g2p\") pod \"4e407044-cf41-4488-a6eb-720ea25a1e4f\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " Nov 02 20:35:06.168208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.168009 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-openshift-ca\") pod \"4e407044-cf41-4488-a6eb-720ea25a1e4f\" (UID: \"4e407044-cf41-4488-a6eb-720ea25a1e4f\") " Nov 02 20:35:06.168482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.168455 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-openshift-ca" pod "4e407044-cf41-4488-a6eb-720ea25a1e4f" (UID: "4e407044-cf41-4488-a6eb-720ea25a1e4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:06.170097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.170076 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-xks-ca" pod "4e407044-cf41-4488-a6eb-720ea25a1e4f" (UID: "4e407044-cf41-4488-a6eb-720ea25a1e4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:06.170169 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.170099 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e407044-cf41-4488-a6eb-720ea25a1e4f-kube-api-access-d9g2p" pod "4e407044-cf41-4488-a6eb-720ea25a1e4f" (UID: "4e407044-cf41-4488-a6eb-720ea25a1e4f"). InnerVolumeSpecName "kube-api-access-d9g2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:06.269381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.269343 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:06.269381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.269373 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e407044-cf41-4488-a6eb-720ea25a1e4f-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:06.269381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.269383 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9g2p\" (UniqueName: \"kubernetes.io/projected/4e407044-cf41-4488-a6eb-720ea25a1e4f-kube-api-access-d9g2p\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:06.913487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.913465 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" Nov 02 20:35:06.913670 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.913456 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd" event={"ID":"4e407044-cf41-4488-a6eb-720ea25a1e4f","Type":"ContainerDied","Data":"459c5d96547765ec9ebef9dd0ab10da97ee70bd1a22fcad3752175b70de9ef9c"} Nov 02 20:35:06.913670 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:06.913587 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="459c5d96547765ec9ebef9dd0ab10da97ee70bd1a22fcad3752175b70de9ef9c" Nov 02 20:35:10.483645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:10.483611 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn"] Nov 02 20:35:10.484147 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:10.483784 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" containerID="cri-o://09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1" gracePeriod=30 Nov 02 20:35:10.496759 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:10.496728 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb"] Nov 02 20:35:10.496971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:10.496943 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" containerName="istio-proxy" containerID="cri-o://44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84" gracePeriod=30 Nov 02 20:35:11.075505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:11.075463 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf"] Nov 02 20:35:11.076496 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:11.076433 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5fc50d-pk5sf"] Nov 02 20:35:11.507650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:11.507573 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85848fb-eb58-42ad-8598-57b13cdadd84" path="/var/lib/kubelet/pods/d85848fb-eb58-42ad-8598-57b13cdadd84/volumes" Nov 02 20:35:15.782505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.782418 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:35:15.823990 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.823964 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:35:15.905120 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905088 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-socket\") pod \"44434a4f-a23d-48fd-a593-b8aee99aaecc\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " Nov 02 20:35:15.905341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905139 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-data\") pod \"1ad575b7-b339-4c48-8119-c971790bbacb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " Nov 02 20:35:15.905341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905166 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-envoy\") pod \"1ad575b7-b339-4c48-8119-c971790bbacb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " Nov 02 20:35:15.905341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905229 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-podinfo\") pod \"44434a4f-a23d-48fd-a593-b8aee99aaecc\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " Nov 02 20:35:15.905341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905265 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-token\" (UniqueName: \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-token\") pod \"44434a4f-a23d-48fd-a593-b8aee99aaecc\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " Nov 02 20:35:15.905341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905317 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-istio-token\" (UniqueName: \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-istio-token\") pod \"1ad575b7-b339-4c48-8119-c971790bbacb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " Nov 02 20:35:15.905628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905425 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-socket" pod "44434a4f-a23d-48fd-a593-b8aee99aaecc" (UID: "44434a4f-a23d-48fd-a593-b8aee99aaecc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.905628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905432 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-credential-socket\") pod \"44434a4f-a23d-48fd-a593-b8aee99aaecc\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " Nov 02 20:35:15.905628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905498 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-data" pod "1ad575b7-b339-4c48-8119-c971790bbacb" (UID: "1ad575b7-b339-4c48-8119-c971790bbacb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.905628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905505 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-envoy\") pod \"44434a4f-a23d-48fd-a593-b8aee99aaecc\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " Nov 02 20:35:15.905805 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905668 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-credential-socket" pod "44434a4f-a23d-48fd-a593-b8aee99aaecc" (UID: "44434a4f-a23d-48fd-a593-b8aee99aaecc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.905944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.905920 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ad575b7-b339-4c48-8119-c971790bbacb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad575b7-b339-4c48-8119-c971790bbacb-istio-podinfo\") pod \"1ad575b7-b339-4c48-8119-c971790bbacb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " Nov 02 20:35:15.906051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.906030 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44434a4f-a23d-48fd-a593-b8aee99aaecc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44434a4f-a23d-48fd-a593-b8aee99aaecc-istiod-ca-cert\") pod \"44434a4f-a23d-48fd-a593-b8aee99aaecc\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " Nov 02 20:35:15.906119 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.906099 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-socket\") pod \"1ad575b7-b339-4c48-8119-c971790bbacb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " Nov 02 20:35:15.906183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.906132 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-data\") pod \"44434a4f-a23d-48fd-a593-b8aee99aaecc\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " Nov 02 20:35:15.906183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.906172 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-certs\") pod \"44434a4f-a23d-48fd-a593-b8aee99aaecc\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " Nov 02 20:35:15.906326 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.906200 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-credential-socket\") pod \"1ad575b7-b339-4c48-8119-c971790bbacb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " Nov 02 20:35:15.906398 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.906339 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad575b7-b339-4c48-8119-c971790bbacb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad575b7-b339-4c48-8119-c971790bbacb-istiod-ca-cert\") pod \"1ad575b7-b339-4c48-8119-c971790bbacb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " Nov 02 20:35:15.906398 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.906385 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-certs\") pod \"1ad575b7-b339-4c48-8119-c971790bbacb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " Nov 02 20:35:15.906512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.906414 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-kube-api-access-lcpmq\" (UniqueName: \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-kube-api-access-lcpmq\") pod \"1ad575b7-b339-4c48-8119-c971790bbacb\" (UID: \"1ad575b7-b339-4c48-8119-c971790bbacb\") " Nov 02 20:35:15.906512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.906487 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-kube-api-access-272gc\" (UniqueName: \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-kube-api-access-272gc\") pod \"44434a4f-a23d-48fd-a593-b8aee99aaecc\" (UID: \"44434a4f-a23d-48fd-a593-b8aee99aaecc\") " Nov 02 20:35:15.906996 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.906970 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44434a4f-a23d-48fd-a593-b8aee99aaecc-istiod-ca-cert" pod "44434a4f-a23d-48fd-a593-b8aee99aaecc" (UID: "44434a4f-a23d-48fd-a593-b8aee99aaecc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:15.907159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.907133 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.907239 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.907166 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.907239 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.907180 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:15.907722 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.907699 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-credential-socket" pod "1ad575b7-b339-4c48-8119-c971790bbacb" (UID: "1ad575b7-b339-4c48-8119-c971790bbacb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.907843 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.907759 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad575b7-b339-4c48-8119-c971790bbacb-istiod-ca-cert" pod "1ad575b7-b339-4c48-8119-c971790bbacb" (UID: "1ad575b7-b339-4c48-8119-c971790bbacb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:15.908030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.907998 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-certs" pod "1ad575b7-b339-4c48-8119-c971790bbacb" (UID: "1ad575b7-b339-4c48-8119-c971790bbacb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.908146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.908051 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-envoy" pod "44434a4f-a23d-48fd-a593-b8aee99aaecc" (UID: "44434a4f-a23d-48fd-a593-b8aee99aaecc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.908146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.908062 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-token" pod "44434a4f-a23d-48fd-a593-b8aee99aaecc" (UID: "44434a4f-a23d-48fd-a593-b8aee99aaecc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:15.908146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.908120 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-socket" pod "1ad575b7-b339-4c48-8119-c971790bbacb" (UID: "1ad575b7-b339-4c48-8119-c971790bbacb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.908369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.908309 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-data" pod "44434a4f-a23d-48fd-a593-b8aee99aaecc" (UID: "44434a4f-a23d-48fd-a593-b8aee99aaecc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.908445 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.908378 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-certs" pod "44434a4f-a23d-48fd-a593-b8aee99aaecc" (UID: "44434a4f-a23d-48fd-a593-b8aee99aaecc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.909062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.909035 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ad575b7-b339-4c48-8119-c971790bbacb-istio-podinfo" pod "1ad575b7-b339-4c48-8119-c971790bbacb" (UID: "1ad575b7-b339-4c48-8119-c971790bbacb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:35:15.909581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.909557 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-envoy" pod "1ad575b7-b339-4c48-8119-c971790bbacb" (UID: "1ad575b7-b339-4c48-8119-c971790bbacb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:15.909694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.909667 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-istio-token" pod "1ad575b7-b339-4c48-8119-c971790bbacb" (UID: "1ad575b7-b339-4c48-8119-c971790bbacb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:15.910359 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.910331 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-kube-api-access-272gc" pod "44434a4f-a23d-48fd-a593-b8aee99aaecc" (UID: "44434a4f-a23d-48fd-a593-b8aee99aaecc"). InnerVolumeSpecName "kube-api-access-272gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:15.910464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.910361 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-kube-api-access-lcpmq" pod "1ad575b7-b339-4c48-8119-c971790bbacb" (UID: "1ad575b7-b339-4c48-8119-c971790bbacb"). InnerVolumeSpecName "kube-api-access-lcpmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:15.910917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.910894 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-podinfo" pod "44434a4f-a23d-48fd-a593-b8aee99aaecc" (UID: "44434a4f-a23d-48fd-a593-b8aee99aaecc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:35:15.981022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.980987 5399 generic.go:356] "Generic (PLEG): container finished" podID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerID="09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1" exitCode=0 Nov 02 20:35:15.981234 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.981070 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" event={"ID":"44434a4f-a23d-48fd-a593-b8aee99aaecc","Type":"ContainerDied","Data":"09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1"} Nov 02 20:35:15.981234 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.981135 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" event={"ID":"44434a4f-a23d-48fd-a593-b8aee99aaecc","Type":"ContainerDied","Data":"8aca8c2b2f22550252fee44ef0cea92b3ce863ea97947b8f7e396cc16a8eaa1e"} Nov 02 20:35:15.981234 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.981156 5399 scope.go:122] "RemoveContainer" containerID="09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1" Nov 02 20:35:15.981417 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.981267 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn" Nov 02 20:35:15.983307 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.983251 5399 generic.go:356] "Generic (PLEG): container finished" podID="1ad575b7-b339-4c48-8119-c971790bbacb" containerID="44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84" exitCode=0 Nov 02 20:35:15.983439 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.983329 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" Nov 02 20:35:15.983439 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.983321 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" event={"ID":"1ad575b7-b339-4c48-8119-c971790bbacb","Type":"ContainerDied","Data":"44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84"} Nov 02 20:35:15.983550 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.983457 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb" event={"ID":"1ad575b7-b339-4c48-8119-c971790bbacb","Type":"ContainerDied","Data":"10c057bb14a6bf7e87dc3b3dda3ef33ea86860ae179355b580ff9c8ba3c00c06"} Nov 02 20:35:15.992423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:15.992397 5399 scope.go:122] "RemoveContainer" containerID="05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f" Nov 02 20:35:16.006984 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.006887 5399 scope.go:122] "RemoveContainer" containerID="09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1" Nov 02 20:35:16.008388 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008345 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44434a4f-a23d-48fd-a593-b8aee99aaecc-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008388 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008381 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008397 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008410 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008424 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008438 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad575b7-b339-4c48-8119-c971790bbacb-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008450 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008464 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcpmq\" (UniqueName: \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-kube-api-access-lcpmq\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008480 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-272gc\" (UniqueName: \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-kube-api-access-272gc\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008493 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad575b7-b339-4c48-8119-c971790bbacb-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008506 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008519 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008532 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad575b7-b339-4c48-8119-c971790bbacb-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008547 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44434a4f-a23d-48fd-a593-b8aee99aaecc-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.008581 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008560 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad575b7-b339-4c48-8119-c971790bbacb-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:16.009177 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:35:16.008902 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1\": container with ID starting with 09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1 not found: ID does not exist" containerID="09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1" Nov 02 20:35:16.009177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008943 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1"} err="failed to get container status \"09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1\": rpc error: code = NotFound desc = could not find container \"09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1\": container with ID starting with 09751170bc3efb6bad885e8e2886ac3e6b60796110cee1a8c431e3d0620a91b1 not found: ID does not exist" Nov 02 20:35:16.009177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.008969 5399 scope.go:122] "RemoveContainer" containerID="05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f" Nov 02 20:35:16.009369 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:35:16.009323 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f\": container with ID starting with 05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f not found: ID does not exist" containerID="05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f" Nov 02 20:35:16.009369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.009347 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f"} err="failed to get container status \"05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f\": rpc error: code = NotFound desc = could not find container \"05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f\": container with ID starting with 05a3fbdf1ac41e4fc37ee85cdf887a9636acf9fdda50412db87a42e91726b88f not found: ID does not exist" Nov 02 20:35:16.009369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.009362 5399 scope.go:122] "RemoveContainer" containerID="44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84" Nov 02 20:35:16.021797 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.021768 5399 scope.go:122] "RemoveContainer" containerID="44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84" Nov 02 20:35:16.022091 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:35:16.022071 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84\": container with ID starting with 44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84 not found: ID does not exist" containerID="44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84" Nov 02 20:35:16.022167 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.022102 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84"} err="failed to get container status \"44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84\": rpc error: code = NotFound desc = could not find container \"44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84\": container with ID starting with 44be2ae91e41334986a94a621717ce83e2fac73df64be9d1a7520a9c32715f84 not found: ID does not exist" Nov 02 20:35:16.111061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.111009 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn"] Nov 02 20:35:16.114314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.114265 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-982b8a-openshift-default-6c964b646f-7k7dn"] Nov 02 20:35:16.220484 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.220433 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb"] Nov 02 20:35:16.223498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:16.223474 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2052e3-openshift-default-5d6484d7d5-bqldb"] Nov 02 20:35:17.395666 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:17.395631 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj"] Nov 02 20:35:17.396104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:17.395883 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" containerID="cri-o://ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5" gracePeriod=30 Nov 02 20:35:17.531587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:17.531547 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" path="/var/lib/kubelet/pods/1ad575b7-b339-4c48-8119-c971790bbacb/volumes" Nov 02 20:35:17.532207 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:17.532185 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" path="/var/lib/kubelet/pods/44434a4f-a23d-48fd-a593-b8aee99aaecc/volumes" Nov 02 20:35:18.227788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.227744 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g"] Nov 02 20:35:18.228573 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228551 5399 cpu_manager.go:414] "containerMap: removing container" podUID="4e407044-cf41-4488-a6eb-720ea25a1e4f" containerName="revoke-keys" Nov 02 20:35:18.228668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228577 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e407044-cf41-4488-a6eb-720ea25a1e4f" containerName="revoke-keys" Nov 02 20:35:18.228668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228605 5399 cpu_manager.go:414] "containerMap: removing container" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" Nov 02 20:35:18.228668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228614 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" Nov 02 20:35:18.228668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228642 5399 cpu_manager.go:414] "containerMap: removing container" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" Nov 02 20:35:18.228668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228652 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" Nov 02 20:35:18.228920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228686 5399 cpu_manager.go:414] "containerMap: removing container" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" containerName="istio-proxy" Nov 02 20:35:18.228920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228695 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" containerName="istio-proxy" Nov 02 20:35:18.228920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228838 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad575b7-b339-4c48-8119-c971790bbacb" containerName="istio-proxy" Nov 02 20:35:18.228920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228858 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e407044-cf41-4488-a6eb-720ea25a1e4f" containerName="revoke-keys" Nov 02 20:35:18.228920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228869 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" Nov 02 20:35:18.228920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.228883 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="44434a4f-a23d-48fd-a593-b8aee99aaecc" containerName="istio-proxy" Nov 02 20:35:18.234420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.234393 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.241355 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:35:18.241327 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:35:18.283221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.282923 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g"] Nov 02 20:35:18.283221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.283006 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:35:18.283221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.283097 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:35:18.336781 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.336748 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bwq2\" (UniqueName: \"kubernetes.io/projected/8eb2278e-8b64-4801-b246-d358d1f2017e-kube-api-access-9bwq2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.336985 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.336934 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.337134 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.337043 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.438209 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.438174 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.438721 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.438236 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.438721 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.438370 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bwq2\" (UniqueName: \"kubernetes.io/projected/8eb2278e-8b64-4801-b246-d358d1f2017e-kube-api-access-9bwq2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.438978 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.438956 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.440817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.440791 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.491553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.491475 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bwq2\" (UniqueName: \"kubernetes.io/projected/8eb2278e-8b64-4801-b246-d358d1f2017e-kube-api-access-9bwq2\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.548932 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.548898 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:18.915320 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:18.915290 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g"] Nov 02 20:35:18.916198 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:35:18.916171 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb2278e_8b64_4801_b246_d358d1f2017e.slice/crio-cc7d2dbb9d373d61480f18af6677eb1c70c7e4dba0e19131c120cf571272b983 WatchSource:0}: Error finding container cc7d2dbb9d373d61480f18af6677eb1c70c7e4dba0e19131c120cf571272b983: Status 404 returned error can't find the container with id cc7d2dbb9d373d61480f18af6677eb1c70c7e4dba0e19131c120cf571272b983 Nov 02 20:35:19.015770 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:19.015731 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" event={"ID":"8eb2278e-8b64-4801-b246-d358d1f2017e","Type":"ContainerStarted","Data":"cc7d2dbb9d373d61480f18af6677eb1c70c7e4dba0e19131c120cf571272b983"} Nov 02 20:35:20.025247 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:20.025215 5399 generic.go:356] "Generic (PLEG): container finished" podID="8eb2278e-8b64-4801-b246-d358d1f2017e" containerID="fb8e3ccb40c89cc2b909a7c94a150bb8714c625635a915e92780ec704ff41110" exitCode=0 Nov 02 20:35:20.025657 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:20.025307 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" event={"ID":"8eb2278e-8b64-4801-b246-d358d1f2017e","Type":"ContainerDied","Data":"fb8e3ccb40c89cc2b909a7c94a150bb8714c625635a915e92780ec704ff41110"} Nov 02 20:35:21.176976 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:21.176951 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:21.272513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:21.272480 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-openshift-ca\") pod \"8eb2278e-8b64-4801-b246-d358d1f2017e\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " Nov 02 20:35:21.272710 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:21.272568 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-xks-ca\") pod \"8eb2278e-8b64-4801-b246-d358d1f2017e\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " Nov 02 20:35:21.272710 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:21.272670 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb2278e-8b64-4801-b246-d358d1f2017e-kube-api-access-9bwq2\" (UniqueName: \"kubernetes.io/projected/8eb2278e-8b64-4801-b246-d358d1f2017e-kube-api-access-9bwq2\") pod \"8eb2278e-8b64-4801-b246-d358d1f2017e\" (UID: \"8eb2278e-8b64-4801-b246-d358d1f2017e\") " Nov 02 20:35:21.272831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:21.272792 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-openshift-ca" pod "8eb2278e-8b64-4801-b246-d358d1f2017e" (UID: "8eb2278e-8b64-4801-b246-d358d1f2017e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:21.273069 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:21.273049 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:21.274662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:21.274637 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-xks-ca" pod "8eb2278e-8b64-4801-b246-d358d1f2017e" (UID: "8eb2278e-8b64-4801-b246-d358d1f2017e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:35:21.274820 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:21.274799 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb2278e-8b64-4801-b246-d358d1f2017e-kube-api-access-9bwq2" pod "8eb2278e-8b64-4801-b246-d358d1f2017e" (UID: "8eb2278e-8b64-4801-b246-d358d1f2017e"). InnerVolumeSpecName "kube-api-access-9bwq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:21.374669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:21.374624 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bwq2\" (UniqueName: \"kubernetes.io/projected/8eb2278e-8b64-4801-b246-d358d1f2017e-kube-api-access-9bwq2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:21.374669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:21.374669 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb2278e-8b64-4801-b246-d358d1f2017e-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:22.042985 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.042951 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" Nov 02 20:35:22.042985 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.042980 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g" event={"ID":"8eb2278e-8b64-4801-b246-d358d1f2017e","Type":"ContainerDied","Data":"cc7d2dbb9d373d61480f18af6677eb1c70c7e4dba0e19131c120cf571272b983"} Nov 02 20:35:22.043256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.043025 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc7d2dbb9d373d61480f18af6677eb1c70c7e4dba0e19131c120cf571272b983" Nov 02 20:35:22.670804 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.670745 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:35:22.756189 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.756156 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:35:22.790322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790290 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istiod-ca-cert\") pod \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " Nov 02 20:35:22.790489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790386 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-envoy\") pod \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " Nov 02 20:35:22.790489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790404 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-certs\") pod \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " Nov 02 20:35:22.790489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790427 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-token\" (UniqueName: \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-token\") pod \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " Nov 02 20:35:22.790489 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790472 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-credential-socket\") pod \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " Nov 02 20:35:22.790686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790526 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-socket\") pod \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " Nov 02 20:35:22.790686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790585 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-podinfo\") pod \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " Nov 02 20:35:22.790686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790613 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-kube-api-access-bq6l5\" (UniqueName: \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-kube-api-access-bq6l5\") pod \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " Nov 02 20:35:22.790686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790621 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istiod-ca-cert" pod "fe65bde2-a8b8-472b-bc2f-4a518e2702de" (UID: "fe65bde2-a8b8-472b-bc2f-4a518e2702de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:35:22.790686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790652 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-data\") pod \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\" (UID: \"fe65bde2-a8b8-472b-bc2f-4a518e2702de\") " Nov 02 20:35:22.790947 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790747 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-certs" pod "fe65bde2-a8b8-472b-bc2f-4a518e2702de" (UID: "fe65bde2-a8b8-472b-bc2f-4a518e2702de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:22.790947 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790820 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-socket" pod "fe65bde2-a8b8-472b-bc2f-4a518e2702de" (UID: "fe65bde2-a8b8-472b-bc2f-4a518e2702de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:22.790947 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.790837 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-credential-socket" pod "fe65bde2-a8b8-472b-bc2f-4a518e2702de" (UID: "fe65bde2-a8b8-472b-bc2f-4a518e2702de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:22.791097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.791043 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-data" pod "fe65bde2-a8b8-472b-bc2f-4a518e2702de" (UID: "fe65bde2-a8b8-472b-bc2f-4a518e2702de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:22.791229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.791210 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:22.791324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.791234 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:22.791324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.791248 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:22.791324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.791263 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:22.791324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.791297 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:22.792762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.792740 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-token" pod "fe65bde2-a8b8-472b-bc2f-4a518e2702de" (UID: "fe65bde2-a8b8-472b-bc2f-4a518e2702de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:22.792840 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.792757 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-podinfo" pod "fe65bde2-a8b8-472b-bc2f-4a518e2702de" (UID: "fe65bde2-a8b8-472b-bc2f-4a518e2702de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:35:22.793043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.793028 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-envoy" pod "fe65bde2-a8b8-472b-bc2f-4a518e2702de" (UID: "fe65bde2-a8b8-472b-bc2f-4a518e2702de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:35:22.793096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.793072 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-kube-api-access-bq6l5" pod "fe65bde2-a8b8-472b-bc2f-4a518e2702de" (UID: "fe65bde2-a8b8-472b-bc2f-4a518e2702de"). InnerVolumeSpecName "kube-api-access-bq6l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:35:22.892793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.892759 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:22.892793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.892787 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:22.892793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.892796 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe65bde2-a8b8-472b-bc2f-4a518e2702de-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:22.893082 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:22.892806 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6l5\" (UniqueName: \"kubernetes.io/projected/fe65bde2-a8b8-472b-bc2f-4a518e2702de-kube-api-access-bq6l5\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:35:23.051764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.051678 5399 generic.go:356] "Generic (PLEG): container finished" podID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerID="ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5" exitCode=0 Nov 02 20:35:23.051764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.051722 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" Nov 02 20:35:23.051993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.051764 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" event={"ID":"fe65bde2-a8b8-472b-bc2f-4a518e2702de","Type":"ContainerDied","Data":"ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5"} Nov 02 20:35:23.051993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.051825 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj" event={"ID":"fe65bde2-a8b8-472b-bc2f-4a518e2702de","Type":"ContainerDied","Data":"989ecfd89ac7bb5b4ad32fabd397a5d6e759e0203496e9070a019a09a2c338f4"} Nov 02 20:35:23.051993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.051855 5399 scope.go:122] "RemoveContainer" containerID="ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5" Nov 02 20:35:23.063200 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.063182 5399 scope.go:122] "RemoveContainer" containerID="703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8" Nov 02 20:35:23.072711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.072690 5399 scope.go:122] "RemoveContainer" containerID="ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5" Nov 02 20:35:23.072993 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:35:23.072974 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5\": container with ID starting with ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5 not found: ID does not exist" containerID="ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5" Nov 02 20:35:23.073061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.073002 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5"} err="failed to get container status \"ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5\": rpc error: code = NotFound desc = could not find container \"ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5\": container with ID starting with ae92c53760c21ed47d336c3d21e90208f2da3319a6678308ad0045b3e68a40b5 not found: ID does not exist" Nov 02 20:35:23.073061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.073020 5399 scope.go:122] "RemoveContainer" containerID="703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8" Nov 02 20:35:23.073267 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:35:23.073249 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8\": container with ID starting with 703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8 not found: ID does not exist" containerID="703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8" Nov 02 20:35:23.073352 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.073288 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8"} err="failed to get container status \"703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8\": rpc error: code = NotFound desc = could not find container \"703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8\": container with ID starting with 703e305e8871d69a07bd46a033a24948d0631f09efdb68252556a00a92f558b8 not found: ID does not exist" Nov 02 20:35:23.205776 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.205743 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj"] Nov 02 20:35:23.210988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.210953 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5fc50d-openshift-default-c6bd6fd45-n5kcj"] Nov 02 20:35:23.507977 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:23.507945 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" path="/var/lib/kubelet/pods/fe65bde2-a8b8-472b-bc2f-4a518e2702de/volumes" Nov 02 20:35:27.913404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.913366 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh"] Nov 02 20:35:27.914152 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.914129 5399 cpu_manager.go:414] "containerMap: removing container" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" Nov 02 20:35:27.914250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.914154 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" Nov 02 20:35:27.914250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.914188 5399 cpu_manager.go:414] "containerMap: removing container" podUID="8eb2278e-8b64-4801-b246-d358d1f2017e" containerName="revoke-keys" Nov 02 20:35:27.914250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.914197 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb2278e-8b64-4801-b246-d358d1f2017e" containerName="revoke-keys" Nov 02 20:35:27.914250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.914243 5399 cpu_manager.go:414] "containerMap: removing container" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" Nov 02 20:35:27.914492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.914251 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" Nov 02 20:35:27.914492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.914415 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" Nov 02 20:35:27.914492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.914437 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb2278e-8b64-4801-b246-d358d1f2017e" containerName="revoke-keys" Nov 02 20:35:27.919373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.919352 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:27.968152 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:27.968117 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-aaa3e7-openshift-default-dockercfg-dnsvg\"" Nov 02 20:35:28.018144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.018111 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh"] Nov 02 20:35:28.052130 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.052094 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-credential-socket\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.052336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.052139 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-certs\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.052336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.052164 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6debae9d-b9fb-40b8-9863-76e329edfdaa-istiod-ca-cert\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.052336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.052268 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-data\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.052478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.052368 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-envoy\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.052478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.052415 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swkhs\" (UniqueName: \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-kube-api-access-swkhs\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.052617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.052522 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-podinfo\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.052617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.052570 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-socket\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.052617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.052589 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-token\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153234 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153203 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-socket\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153234 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153243 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-token\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153319 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-credential-socket\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153352 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-certs\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153487 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6debae9d-b9fb-40b8-9863-76e329edfdaa-istiod-ca-cert\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153551 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-data\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153629 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-envoy\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153805 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153679 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-socket\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153805 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153698 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swkhs\" (UniqueName: \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-kube-api-access-swkhs\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153805 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153738 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-credential-socket\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.153805 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153782 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-certs\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.154012 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153819 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-podinfo\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.154012 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.153940 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-data\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.154230 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.154204 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6debae9d-b9fb-40b8-9863-76e329edfdaa-istiod-ca-cert\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.156096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.156071 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-envoy\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.156414 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.156394 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-podinfo\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.206716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.206655 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swkhs\" (UniqueName: \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-kube-api-access-swkhs\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.206855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.206838 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-token\") pod \"e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.233180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.233150 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:28.649568 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.649539 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh"] Nov 02 20:35:28.650365 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:35:28.650338 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6debae9d_b9fb_40b8_9863_76e329edfdaa.slice/crio-776d94e0ea11110c1ae85ef562f6987b82cf15ffc604e674ddf3f5468877ecf5 WatchSource:0}: Error finding container 776d94e0ea11110c1ae85ef562f6987b82cf15ffc604e674ddf3f5468877ecf5: Status 404 returned error can't find the container with id 776d94e0ea11110c1ae85ef562f6987b82cf15ffc604e674ddf3f5468877ecf5 Nov 02 20:35:28.653068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.653033 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:28.653145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.653100 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:28.653145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:28.653128 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:29.102151 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:29.102110 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" event={"ID":"6debae9d-b9fb-40b8-9863-76e329edfdaa","Type":"ContainerStarted","Data":"45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19"} Nov 02 20:35:29.102612 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:29.102171 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" event={"ID":"6debae9d-b9fb-40b8-9863-76e329edfdaa","Type":"ContainerStarted","Data":"776d94e0ea11110c1ae85ef562f6987b82cf15ffc604e674ddf3f5468877ecf5"} Nov 02 20:35:29.212798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:29.212742 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" podStartSLOduration=2.212726403 podStartE2EDuration="2.212726403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:29.210102765 +0000 UTC m=+2571.820754868" watchObservedRunningTime="2026-11-02 20:35:29.212726403 +0000 UTC m=+2571.823378495" Nov 02 20:35:29.234045 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:29.234011 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:29.240386 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:29.240362 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:30.114210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:30.114180 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:30.115241 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:30.115217 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:35:38.814596 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:38.814563 5399 scope.go:122] "RemoveContainer" containerID="4c7f2480b4911dc7af2137964319458334056d2fc237d893e8c96cb1ea2b07d6" Nov 02 20:35:38.825164 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:38.825138 5399 scope.go:122] "RemoveContainer" containerID="13159b83946569ad8f18f7a0d01c40f63c591be828e76367de8c410f2fe38045" Nov 02 20:35:39.012846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.012808 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl"] Nov 02 20:35:39.013592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.013568 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe65bde2-a8b8-472b-bc2f-4a518e2702de" containerName="istio-proxy" Nov 02 20:35:39.019004 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.018979 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:35:39.067227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.067150 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl"] Nov 02 20:35:39.094325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.094284 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1b1358-8da7-4ee0-afae-fa9af7e35144-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl\" (UID: \"3b1b1358-8da7-4ee0-afae-fa9af7e35144\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:35:39.094471 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.094401 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtkcc\" (UniqueName: \"kubernetes.io/projected/3b1b1358-8da7-4ee0-afae-fa9af7e35144-kube-api-access-vtkcc\") pod \"payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl\" (UID: \"3b1b1358-8da7-4ee0-afae-fa9af7e35144\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:35:39.109093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.109063 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-aaa3e7\"" Nov 02 20:35:39.109093 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.109062 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-aaa3e7-dockercfg-bnpn6\"" Nov 02 20:35:39.195417 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.195379 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1b1358-8da7-4ee0-afae-fa9af7e35144-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl\" (UID: \"3b1b1358-8da7-4ee0-afae-fa9af7e35144\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:35:39.195601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.195457 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkcc\" (UniqueName: \"kubernetes.io/projected/3b1b1358-8da7-4ee0-afae-fa9af7e35144-kube-api-access-vtkcc\") pod \"payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl\" (UID: \"3b1b1358-8da7-4ee0-afae-fa9af7e35144\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:35:39.196129 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.196108 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1b1358-8da7-4ee0-afae-fa9af7e35144-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl\" (UID: \"3b1b1358-8da7-4ee0-afae-fa9af7e35144\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:35:39.248375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.248347 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkcc\" (UniqueName: \"kubernetes.io/projected/3b1b1358-8da7-4ee0-afae-fa9af7e35144-kube-api-access-vtkcc\") pod \"payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl\" (UID: \"3b1b1358-8da7-4ee0-afae-fa9af7e35144\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:35:39.333500 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.333397 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:35:39.713305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:39.713247 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl"] Nov 02 20:35:39.713634 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:35:39.713612 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1b1358_8da7_4ee0_afae_fa9af7e35144.slice/crio-0b0faa66b1a7966b80eab79d133bbbb194e0b232d2f4b582b61341a7e04b0fe6 WatchSource:0}: Error finding container 0b0faa66b1a7966b80eab79d133bbbb194e0b232d2f4b582b61341a7e04b0fe6: Status 404 returned error can't find the container with id 0b0faa66b1a7966b80eab79d133bbbb194e0b232d2f4b582b61341a7e04b0fe6 Nov 02 20:35:40.201910 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:40.201871 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" event={"ID":"3b1b1358-8da7-4ee0-afae-fa9af7e35144","Type":"ContainerStarted","Data":"f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e"} Nov 02 20:35:40.202350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:40.201927 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" event={"ID":"3b1b1358-8da7-4ee0-afae-fa9af7e35144","Type":"ContainerStarted","Data":"0b0faa66b1a7966b80eab79d133bbbb194e0b232d2f4b582b61341a7e04b0fe6"} Nov 02 20:35:40.202350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:40.201957 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:35:40.310232 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:40.310153 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" podStartSLOduration=2.310136569 podStartE2EDuration="2.310136569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:40.310051248 +0000 UTC m=+2582.920703341" watchObservedRunningTime="2026-11-02 20:35:40.310136569 +0000 UTC m=+2582.920788661" Nov 02 20:35:48.237682 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:35:48.237641 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:35:51.211470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:51.211437 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:35:52.299327 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:52.299289 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:35:56.030338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:56.030256 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl"] Nov 02 20:35:56.076935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:56.076905 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-accdf5-q9btl"] Nov 02 20:35:57.509599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:35:57.509568 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33890af3-f7cb-4a17-99aa-24ed74d901cb" path="/var/lib/kubelet/pods/33890af3-f7cb-4a17-99aa-24ed74d901cb/volumes" Nov 02 20:36:02.341447 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:02.341406 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl"] Nov 02 20:36:02.342080 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:02.341685 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" containerID="cri-o://3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478" gracePeriod=30 Nov 02 20:36:07.614997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.614927 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:36:07.644726 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.644696 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istiod-ca-cert\") pod \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " Nov 02 20:36:07.644869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.644754 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-podinfo\") pod \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " Nov 02 20:36:07.644869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.644822 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-data\") pod \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " Nov 02 20:36:07.644869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.644845 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-socket\") pod \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " Nov 02 20:36:07.644869 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.644864 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-envoy\") pod \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " Nov 02 20:36:07.645053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.644884 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-credential-socket\") pod \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " Nov 02 20:36:07.645053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.644919 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-token\" (UniqueName: \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-token\") pod \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " Nov 02 20:36:07.645053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.644960 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-kube-api-access-jhslw\" (UniqueName: \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-kube-api-access-jhslw\") pod \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " Nov 02 20:36:07.645209 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.645067 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istiod-ca-cert" pod "439a19f0-c499-4ef7-a96c-ce8f6ee07df8" (UID: "439a19f0-c499-4ef7-a96c-ce8f6ee07df8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:07.645372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.645262 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-data" pod "439a19f0-c499-4ef7-a96c-ce8f6ee07df8" (UID: "439a19f0-c499-4ef7-a96c-ce8f6ee07df8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:07.645372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.645319 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-certs\") pod \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\" (UID: \"439a19f0-c499-4ef7-a96c-ce8f6ee07df8\") " Nov 02 20:36:07.645372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.645326 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-socket" pod "439a19f0-c499-4ef7-a96c-ce8f6ee07df8" (UID: "439a19f0-c499-4ef7-a96c-ce8f6ee07df8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:07.645372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.645347 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-credential-socket" pod "439a19f0-c499-4ef7-a96c-ce8f6ee07df8" (UID: "439a19f0-c499-4ef7-a96c-ce8f6ee07df8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:07.645856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.645830 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-certs" pod "439a19f0-c499-4ef7-a96c-ce8f6ee07df8" (UID: "439a19f0-c499-4ef7-a96c-ce8f6ee07df8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:07.645981 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.645886 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:07.645981 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.645903 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:07.645981 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.645913 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:07.645981 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.645924 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:07.647341 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.647320 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-token" pod "439a19f0-c499-4ef7-a96c-ce8f6ee07df8" (UID: "439a19f0-c499-4ef7-a96c-ce8f6ee07df8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:07.647431 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.647313 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-envoy" pod "439a19f0-c499-4ef7-a96c-ce8f6ee07df8" (UID: "439a19f0-c499-4ef7-a96c-ce8f6ee07df8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:07.647527 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.647507 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-podinfo" pod "439a19f0-c499-4ef7-a96c-ce8f6ee07df8" (UID: "439a19f0-c499-4ef7-a96c-ce8f6ee07df8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:36:07.647578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.647555 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-kube-api-access-jhslw" pod "439a19f0-c499-4ef7-a96c-ce8f6ee07df8" (UID: "439a19f0-c499-4ef7-a96c-ce8f6ee07df8"). InnerVolumeSpecName "kube-api-access-jhslw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:07.747355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.747319 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:07.747355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.747348 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:07.747355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.747359 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:07.747598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.747369 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:07.747598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:07.747379 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhslw\" (UniqueName: \"kubernetes.io/projected/439a19f0-c499-4ef7-a96c-ce8f6ee07df8-kube-api-access-jhslw\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:08.396847 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:08.396805 5399 generic.go:356] "Generic (PLEG): container finished" podID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerID="3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478" exitCode=0 Nov 02 20:36:08.397028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:08.396850 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" Nov 02 20:36:08.397028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:08.396878 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" event={"ID":"439a19f0-c499-4ef7-a96c-ce8f6ee07df8","Type":"ContainerDied","Data":"3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478"} Nov 02 20:36:08.397028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:08.396939 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl" event={"ID":"439a19f0-c499-4ef7-a96c-ce8f6ee07df8","Type":"ContainerDied","Data":"c2332957471fe5948dc3b692f3626d9896405f23702af85036d0ce64c4e4a4c7"} Nov 02 20:36:08.397028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:08.396972 5399 scope.go:122] "RemoveContainer" containerID="3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478" Nov 02 20:36:08.407574 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:08.407551 5399 scope.go:122] "RemoveContainer" containerID="3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478" Nov 02 20:36:08.407828 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:36:08.407810 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478\": container with ID starting with 3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478 not found: ID does not exist" containerID="3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478" Nov 02 20:36:08.407878 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:08.407835 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478"} err="failed to get container status \"3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478\": rpc error: code = NotFound desc = could not find container \"3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478\": container with ID starting with 3e81190232d6e6b88e4540b89e05a0d40b81a9fb52f917d6dbe93839c36f6478 not found: ID does not exist" Nov 02 20:36:08.551332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:08.551299 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl"] Nov 02 20:36:08.597345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:08.597301 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-accdf5-openshift-default-5d8f9ddbf5-rmccl"] Nov 02 20:36:09.506964 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:09.506930 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" path="/var/lib/kubelet/pods/439a19f0-c499-4ef7-a96c-ce8f6ee07df8/volumes" Nov 02 20:36:11.773311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:11.773263 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd"] Nov 02 20:36:11.819619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:11.819581 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5efe19-hcbjd"] Nov 02 20:36:13.508063 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:13.508031 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e407044-cf41-4488-a6eb-720ea25a1e4f" path="/var/lib/kubelet/pods/4e407044-cf41-4488-a6eb-720ea25a1e4f/volumes" Nov 02 20:36:18.239871 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:36:18.239770 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:36:19.203540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.203497 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz"] Nov 02 20:36:19.204110 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.204090 5399 cpu_manager.go:414] "containerMap: removing container" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" Nov 02 20:36:19.204178 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.204112 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" Nov 02 20:36:19.204234 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.204218 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="439a19f0-c499-4ef7-a96c-ce8f6ee07df8" containerName="istio-proxy" Nov 02 20:36:19.209125 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.209103 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.259970 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.259930 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz"] Nov 02 20:36:19.279702 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.279663 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.279881 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.279724 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klf6j\" (UniqueName: \"kubernetes.io/projected/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-kube-api-access-klf6j\") pod \"maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.279936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.279908 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.302652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.302616 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:36:19.302817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.302711 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:36:19.380800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.380766 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.381007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.380815 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klf6j\" (UniqueName: \"kubernetes.io/projected/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-kube-api-access-klf6j\") pod \"maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.381007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.380975 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.381579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.381555 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.383564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.383543 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.434703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.434665 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klf6j\" (UniqueName: \"kubernetes.io/projected/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-kube-api-access-klf6j\") pod \"maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.522705 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.522619 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:19.889910 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:19.889876 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz"] Nov 02 20:36:19.890999 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:36:19.890966 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cc6b53_fcfa_4e44_a4a1_e6f09de5ad13.slice/crio-b862699a0a0e544ca39ab4bf74eafa063dfedee955c405e28ca1a7779a360563 WatchSource:0}: Error finding container b862699a0a0e544ca39ab4bf74eafa063dfedee955c405e28ca1a7779a360563: Status 404 returned error can't find the container with id b862699a0a0e544ca39ab4bf74eafa063dfedee955c405e28ca1a7779a360563 Nov 02 20:36:20.482695 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:20.482599 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" event={"ID":"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13","Type":"ContainerStarted","Data":"9ba49c54e4a12bb57aa1b138e1cc8e6e8f6b8aff32323716b4a930f35a9ed817"} Nov 02 20:36:20.482695 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:20.482669 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" event={"ID":"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13","Type":"ContainerStarted","Data":"b862699a0a0e544ca39ab4bf74eafa063dfedee955c405e28ca1a7779a360563"} Nov 02 20:36:20.589577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:20.589520 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" podStartSLOduration=1.5895020880000001 podStartE2EDuration="1.589502088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:20.586788195 +0000 UTC m=+2623.197440286" watchObservedRunningTime="2026-11-02 20:36:20.589502088 +0000 UTC m=+2623.200154179" Nov 02 20:36:21.491661 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:21.491630 5399 generic.go:356] "Generic (PLEG): container finished" podID="d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13" containerID="9ba49c54e4a12bb57aa1b138e1cc8e6e8f6b8aff32323716b4a930f35a9ed817" exitCode=0 Nov 02 20:36:21.492054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:21.491719 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" event={"ID":"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13","Type":"ContainerDied","Data":"9ba49c54e4a12bb57aa1b138e1cc8e6e8f6b8aff32323716b4a930f35a9ed817"} Nov 02 20:36:22.645819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:22.645790 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:22.720746 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:22.720713 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-xks-ca\") pod \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " Nov 02 20:36:22.720913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:22.720763 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-openshift-ca\") pod \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " Nov 02 20:36:22.720979 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:22.720945 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-kube-api-access-klf6j\" (UniqueName: \"kubernetes.io/projected/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-kube-api-access-klf6j\") pod \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\" (UID: \"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13\") " Nov 02 20:36:22.721188 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:22.721167 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-openshift-ca" pod "d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13" (UID: "d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:22.721411 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:22.721395 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:22.722923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:22.722896 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-xks-ca" pod "d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13" (UID: "d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:22.723040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:22.722986 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-kube-api-access-klf6j" pod "d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13" (UID: "d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13"). InnerVolumeSpecName "kube-api-access-klf6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:22.822142 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:22.822116 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:22.822325 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:22.822144 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klf6j\" (UniqueName: \"kubernetes.io/projected/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13-kube-api-access-klf6j\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:23.509398 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:23.509370 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" Nov 02 20:36:23.509597 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:23.509370 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz" event={"ID":"d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13","Type":"ContainerDied","Data":"b862699a0a0e544ca39ab4bf74eafa063dfedee955c405e28ca1a7779a360563"} Nov 02 20:36:23.509597 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:23.509553 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b862699a0a0e544ca39ab4bf74eafa063dfedee955c405e28ca1a7779a360563" Nov 02 20:36:23.712014 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:23.711978 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd"] Nov 02 20:36:27.573542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:27.573509 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g"] Nov 02 20:36:27.619671 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:27.619629 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-40425f-hvv4g"] Nov 02 20:36:29.508712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:29.508677 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb2278e-8b64-4801-b246-d358d1f2017e" path="/var/lib/kubelet/pods/8eb2278e-8b64-4801-b246-d358d1f2017e/volumes" Nov 02 20:36:34.043783 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:34.043744 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p"] Nov 02 20:36:34.090456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:34.090418 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ad024e-dk57p"] Nov 02 20:36:35.089360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.089321 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5"] Nov 02 20:36:35.090211 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.090184 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13" containerName="revoke-keys" Nov 02 20:36:35.090405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.090388 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13" containerName="revoke-keys" Nov 02 20:36:35.090711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.090690 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13" containerName="revoke-keys" Nov 02 20:36:35.094308 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.094256 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:35.191499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.191463 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:36:35.191499 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.191479 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:36:35.194616 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.194595 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5"] Nov 02 20:36:35.263107 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.263065 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:35.263319 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.263227 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:35.263376 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.263327 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4r9\" (UniqueName: \"kubernetes.io/projected/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-kube-api-access-tw4r9\") pod \"maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:35.364826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.364739 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:35.365006 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.364845 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:35.365006 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.364887 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4r9\" (UniqueName: \"kubernetes.io/projected/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-kube-api-access-tw4r9\") pod \"maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:35.365616 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.365589 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:35.367114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.367086 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:35.417703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.417676 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4r9\" (UniqueName: \"kubernetes.io/projected/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-kube-api-access-tw4r9\") pod \"maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:35.509398 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.509364 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492a1b11-831e-424b-8620-0e9476094a71" path="/var/lib/kubelet/pods/492a1b11-831e-424b-8620-0e9476094a71/volumes" Nov 02 20:36:35.709950 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:35.709875 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:36.080314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:36.079359 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5"] Nov 02 20:36:36.597701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:36.597615 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" event={"ID":"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e","Type":"ContainerStarted","Data":"3bf0132afae16a225a363d95d290be7da975d247623266f864fada6a38e3f7ee"} Nov 02 20:36:36.597701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:36.597668 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" event={"ID":"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e","Type":"ContainerStarted","Data":"88855b48c7f33c6bea48ea1fa4264ee2f40b789aa083607c61bf436790c051a6"} Nov 02 20:36:36.704598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:36.704545 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" podStartSLOduration=1.704531066 podStartE2EDuration="1.704531066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:36.70255372 +0000 UTC m=+2639.313205812" watchObservedRunningTime="2026-11-02 20:36:36.704531066 +0000 UTC m=+2639.315183157" Nov 02 20:36:37.605412 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:37.605377 5399 generic.go:356] "Generic (PLEG): container finished" podID="3ea2fe64-b1a3-44aa-8f92-d879baa2b40e" containerID="3bf0132afae16a225a363d95d290be7da975d247623266f864fada6a38e3f7ee" exitCode=0 Nov 02 20:36:37.605801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:37.605454 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" event={"ID":"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e","Type":"ContainerDied","Data":"3bf0132afae16a225a363d95d290be7da975d247623266f864fada6a38e3f7ee"} Nov 02 20:36:38.732785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:38.732746 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" containerID="cri-o://2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad" gracePeriod=45 Nov 02 20:36:38.810451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:38.810422 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:38.883332 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:38.883221 5399 scope.go:122] "RemoveContainer" containerID="ae397de6b49e3622e87cd48497aface58c97b02f8188e9bcdd120adbd876fb8c" Nov 02 20:36:38.902591 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:38.902561 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-xks-ca\") pod \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " Nov 02 20:36:38.902749 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:38.902681 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-openshift-ca\") pod \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " Nov 02 20:36:38.902797 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:38.902772 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-kube-api-access-tw4r9\" (UniqueName: \"kubernetes.io/projected/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-kube-api-access-tw4r9\") pod \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\" (UID: \"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e\") " Nov 02 20:36:38.903214 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:38.903179 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-openshift-ca" pod "3ea2fe64-b1a3-44aa-8f92-d879baa2b40e" (UID: "3ea2fe64-b1a3-44aa-8f92-d879baa2b40e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:38.905666 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:38.905631 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-xks-ca" pod "3ea2fe64-b1a3-44aa-8f92-d879baa2b40e" (UID: "3ea2fe64-b1a3-44aa-8f92-d879baa2b40e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:38.912436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:38.912396 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-kube-api-access-tw4r9" pod "3ea2fe64-b1a3-44aa-8f92-d879baa2b40e" (UID: "3ea2fe64-b1a3-44aa-8f92-d879baa2b40e"). InnerVolumeSpecName "kube-api-access-tw4r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:39.003008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.002979 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:36:39.003819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.003794 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.003926 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.003821 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.003926 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.003837 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw4r9\" (UniqueName: \"kubernetes.io/projected/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e-kube-api-access-tw4r9\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.104230 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.104193 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1e6beb-1003-4bb8-bc07-ade27a80afce-kube-api-access-xjjwf\" (UniqueName: \"kubernetes.io/projected/3f1e6beb-1003-4bb8-bc07-ade27a80afce-kube-api-access-xjjwf\") pod \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " Nov 02 20:36:39.104454 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.104305 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-model-cache\") pod \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " Nov 02 20:36:39.104454 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.104358 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-home\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-home\") pod \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " Nov 02 20:36:39.104587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.104451 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tls-certs\") pod \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " Nov 02 20:36:39.104587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.104510 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tmp-dir\") pod \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " Nov 02 20:36:39.104694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.104618 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-model-cache" pod "3f1e6beb-1003-4bb8-bc07-ade27a80afce" (UID: "3f1e6beb-1003-4bb8-bc07-ade27a80afce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:39.104694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.104639 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-home" pod "3f1e6beb-1003-4bb8-bc07-ade27a80afce" (UID: "3f1e6beb-1003-4bb8-bc07-ade27a80afce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:39.104694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.104649 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-dshm\") pod \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\" (UID: \"3f1e6beb-1003-4bb8-bc07-ade27a80afce\") " Nov 02 20:36:39.104860 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.104833 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tmp-dir" pod "3f1e6beb-1003-4bb8-bc07-ade27a80afce" (UID: "3f1e6beb-1003-4bb8-bc07-ade27a80afce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:39.105183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.105156 5399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tmp-dir\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.105310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.105184 5399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-model-cache\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.105310 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.105200 5399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-home\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.107396 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.107365 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-dshm" pod "3f1e6beb-1003-4bb8-bc07-ade27a80afce" (UID: "3f1e6beb-1003-4bb8-bc07-ade27a80afce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:36:39.107396 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.107382 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1e6beb-1003-4bb8-bc07-ade27a80afce-kube-api-access-xjjwf" pod "3f1e6beb-1003-4bb8-bc07-ade27a80afce" (UID: "3f1e6beb-1003-4bb8-bc07-ade27a80afce"). InnerVolumeSpecName "kube-api-access-xjjwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:39.107565 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.107451 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tls-certs" pod "3f1e6beb-1003-4bb8-bc07-ade27a80afce" (UID: "3f1e6beb-1003-4bb8-bc07-ade27a80afce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:39.205858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.205779 5399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f1e6beb-1003-4bb8-bc07-ade27a80afce-tls-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.205858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.205806 5399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f1e6beb-1003-4bb8-bc07-ade27a80afce-dshm\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.205858 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.205817 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjjwf\" (UniqueName: \"kubernetes.io/projected/3f1e6beb-1003-4bb8-bc07-ade27a80afce-kube-api-access-xjjwf\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:39.619790 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.619756 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" Nov 02 20:36:39.619790 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.619780 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5" event={"ID":"3ea2fe64-b1a3-44aa-8f92-d879baa2b40e","Type":"ContainerDied","Data":"88855b48c7f33c6bea48ea1fa4264ee2f40b789aa083607c61bf436790c051a6"} Nov 02 20:36:39.620048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.619828 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88855b48c7f33c6bea48ea1fa4264ee2f40b789aa083607c61bf436790c051a6" Nov 02 20:36:39.621425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.621400 5399 generic.go:356] "Generic (PLEG): container finished" podID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerID="2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad" exitCode=0 Nov 02 20:36:39.621548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.621432 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" event={"ID":"3f1e6beb-1003-4bb8-bc07-ade27a80afce","Type":"ContainerDied","Data":"2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad"} Nov 02 20:36:39.621548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.621437 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" Nov 02 20:36:39.621548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.621455 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd" event={"ID":"3f1e6beb-1003-4bb8-bc07-ade27a80afce","Type":"ContainerDied","Data":"9125462a055409943087a0f8812b78391f33b45edd6c116b1e542de6f28bac1e"} Nov 02 20:36:39.621548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.621473 5399 scope.go:122] "RemoveContainer" containerID="2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad" Nov 02 20:36:39.631792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.631764 5399 scope.go:122] "RemoveContainer" containerID="2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad" Nov 02 20:36:39.632070 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:36:39.632043 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad\": container with ID starting with 2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad not found: ID does not exist" containerID="2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad" Nov 02 20:36:39.632147 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.632083 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad"} err="failed to get container status \"2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad\": rpc error: code = NotFound desc = could not find container \"2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad\": container with ID starting with 2414c5874b29d97e3747b29b53ca3e331835e3e0f28bcb5bcec08a0ef22d47ad not found: ID does not exist" Nov 02 20:36:39.774266 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.774161 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd"] Nov 02 20:36:39.777505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:39.777476 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2052e3/simulator-w3-2052e3-kserve-7cb544c977-wfjcd"] Nov 02 20:36:40.739020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:40.738981 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:36:41.127286 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.127237 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf"] Nov 02 20:36:41.127780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.127766 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3ea2fe64-b1a3-44aa-8f92-d879baa2b40e" containerName="revoke-keys" Nov 02 20:36:41.127829 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.127780 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea2fe64-b1a3-44aa-8f92-d879baa2b40e" containerName="revoke-keys" Nov 02 20:36:41.127829 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.127803 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" Nov 02 20:36:41.127829 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.127808 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" Nov 02 20:36:41.127924 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.127877 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" containerName="main" Nov 02 20:36:41.127924 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.127894 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea2fe64-b1a3-44aa-8f92-d879baa2b40e" containerName="revoke-keys" Nov 02 20:36:41.132715 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.132683 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.184451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.184419 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:36:41.185049 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.185020 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf"] Nov 02 20:36:41.225817 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.225781 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f1dad2-ggcbf\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.226017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.225838 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjhp\" (UniqueName: \"kubernetes.io/projected/e4cf1f20-7f73-4775-b070-1f3ce66c8973-kube-api-access-9wjhp\") pod \"maas-api-revoke-sub-22f1dad2-ggcbf\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.226116 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.226093 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f1dad2-ggcbf\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.226959 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.226937 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:36:41.327257 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.327215 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f1dad2-ggcbf\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.327520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.327307 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjhp\" (UniqueName: \"kubernetes.io/projected/e4cf1f20-7f73-4775-b070-1f3ce66c8973-kube-api-access-9wjhp\") pod \"maas-api-revoke-sub-22f1dad2-ggcbf\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.327520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.327501 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f1dad2-ggcbf\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.328158 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.328129 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f1dad2-ggcbf\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.329953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.329928 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f1dad2-ggcbf\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.381261 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.381184 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjhp\" (UniqueName: \"kubernetes.io/projected/e4cf1f20-7f73-4775-b070-1f3ce66c8973-kube-api-access-9wjhp\") pod \"maas-api-revoke-sub-22f1dad2-ggcbf\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.448122 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.448086 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:41.511627 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.508627 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1e6beb-1003-4bb8-bc07-ade27a80afce" path="/var/lib/kubelet/pods/3f1e6beb-1003-4bb8-bc07-ade27a80afce/volumes" Nov 02 20:36:41.823617 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:41.823578 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf"] Nov 02 20:36:41.825615 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:36:41.825586 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cf1f20_7f73_4775_b070_1f3ce66c8973.slice/crio-7f5a9fb36d85cc8854a963ef90461c02c628fb7f4a5ad55a0b44b6f6a1253df7 WatchSource:0}: Error finding container 7f5a9fb36d85cc8854a963ef90461c02c628fb7f4a5ad55a0b44b6f6a1253df7: Status 404 returned error can't find the container with id 7f5a9fb36d85cc8854a963ef90461c02c628fb7f4a5ad55a0b44b6f6a1253df7 Nov 02 20:36:42.644839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:42.644802 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" event={"ID":"e4cf1f20-7f73-4775-b070-1f3ce66c8973","Type":"ContainerStarted","Data":"ff02f27b39611ec692051a977f3e36c8821d77a31bf3347faaf7bce6105598c3"} Nov 02 20:36:42.645369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:42.644874 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" event={"ID":"e4cf1f20-7f73-4775-b070-1f3ce66c8973","Type":"ContainerStarted","Data":"7f5a9fb36d85cc8854a963ef90461c02c628fb7f4a5ad55a0b44b6f6a1253df7"} Nov 02 20:36:42.751941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:42.751879 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" podStartSLOduration=1.7518643699999998 podStartE2EDuration="1.75186437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:42.75121205 +0000 UTC m=+2645.361864142" watchObservedRunningTime="2026-11-02 20:36:42.75186437 +0000 UTC m=+2645.362516461" Nov 02 20:36:43.652699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:43.652664 5399 generic.go:356] "Generic (PLEG): container finished" podID="e4cf1f20-7f73-4775-b070-1f3ce66c8973" containerID="ff02f27b39611ec692051a977f3e36c8821d77a31bf3347faaf7bce6105598c3" exitCode=0 Nov 02 20:36:43.653148 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:43.652773 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" event={"ID":"e4cf1f20-7f73-4775-b070-1f3ce66c8973","Type":"ContainerDied","Data":"ff02f27b39611ec692051a977f3e36c8821d77a31bf3347faaf7bce6105598c3"} Nov 02 20:36:44.805592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:44.805563 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:44.866462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:44.866426 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-openshift-ca\") pod \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " Nov 02 20:36:44.866645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:44.866529 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-xks-ca\") pod \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " Nov 02 20:36:44.866645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:44.866572 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cf1f20-7f73-4775-b070-1f3ce66c8973-kube-api-access-9wjhp\" (UniqueName: \"kubernetes.io/projected/e4cf1f20-7f73-4775-b070-1f3ce66c8973-kube-api-access-9wjhp\") pod \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\" (UID: \"e4cf1f20-7f73-4775-b070-1f3ce66c8973\") " Nov 02 20:36:44.866808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:44.866782 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-openshift-ca" pod "e4cf1f20-7f73-4775-b070-1f3ce66c8973" (UID: "e4cf1f20-7f73-4775-b070-1f3ce66c8973"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:44.866972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:44.866955 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:44.868674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:44.868650 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-xks-ca" pod "e4cf1f20-7f73-4775-b070-1f3ce66c8973" (UID: "e4cf1f20-7f73-4775-b070-1f3ce66c8973"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:44.868766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:44.868702 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cf1f20-7f73-4775-b070-1f3ce66c8973-kube-api-access-9wjhp" pod "e4cf1f20-7f73-4775-b070-1f3ce66c8973" (UID: "e4cf1f20-7f73-4775-b070-1f3ce66c8973"). InnerVolumeSpecName "kube-api-access-9wjhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:44.967730 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:44.967649 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cf1f20-7f73-4775-b070-1f3ce66c8973-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:44.967730 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:44.967680 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wjhp\" (UniqueName: \"kubernetes.io/projected/e4cf1f20-7f73-4775-b070-1f3ce66c8973-kube-api-access-9wjhp\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:45.676712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:45.676672 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" event={"ID":"e4cf1f20-7f73-4775-b070-1f3ce66c8973","Type":"ContainerDied","Data":"7f5a9fb36d85cc8854a963ef90461c02c628fb7f4a5ad55a0b44b6f6a1253df7"} Nov 02 20:36:45.676712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:45.676705 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf" Nov 02 20:36:45.676931 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:45.676730 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f5a9fb36d85cc8854a963ef90461c02c628fb7f4a5ad55a0b44b6f6a1253df7" Nov 02 20:36:48.236822 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:36:48.236776 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:36:48.325643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:48.325610 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m"] Nov 02 20:36:48.434043 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:48.434011 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx"] Nov 02 20:36:53.326067 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.326023 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" podUID="6677808e-467a-4e97-be61-95b8f53df853" containerName="payload-processing" containerID="cri-o://1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e" gracePeriod=25 Nov 02 20:36:53.435227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.435163 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" podUID="0050aa91-905a-4fc3-835a-9c557c04ccdd" containerName="payload-pre-processing" containerID="cri-o://a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b" gracePeriod=25 Nov 02 20:36:53.594446 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.594379 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:36:53.671367 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.671322 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6677808e-467a-4e97-be61-95b8f53df853-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6677808e-467a-4e97-be61-95b8f53df853-plugins-config-volume\") pod \"6677808e-467a-4e97-be61-95b8f53df853\" (UID: \"6677808e-467a-4e97-be61-95b8f53df853\") " Nov 02 20:36:53.671567 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.671409 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6677808e-467a-4e97-be61-95b8f53df853-kube-api-access-82xb8\" (UniqueName: \"kubernetes.io/projected/6677808e-467a-4e97-be61-95b8f53df853-kube-api-access-82xb8\") pod \"6677808e-467a-4e97-be61-95b8f53df853\" (UID: \"6677808e-467a-4e97-be61-95b8f53df853\") " Nov 02 20:36:53.671828 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.671798 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6677808e-467a-4e97-be61-95b8f53df853-plugins-config-volume" pod "6677808e-467a-4e97-be61-95b8f53df853" (UID: "6677808e-467a-4e97-be61-95b8f53df853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:53.672027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.672008 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6677808e-467a-4e97-be61-95b8f53df853-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:53.673585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.673560 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6677808e-467a-4e97-be61-95b8f53df853-kube-api-access-82xb8" pod "6677808e-467a-4e97-be61-95b8f53df853" (UID: "6677808e-467a-4e97-be61-95b8f53df853"). InnerVolumeSpecName "kube-api-access-82xb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:53.690192 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.690163 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:36:53.735096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.735048 5399 generic.go:356] "Generic (PLEG): container finished" podID="0050aa91-905a-4fc3-835a-9c557c04ccdd" containerID="a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b" exitCode=0 Nov 02 20:36:53.735096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.735094 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" Nov 02 20:36:53.735375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.735133 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" event={"ID":"0050aa91-905a-4fc3-835a-9c557c04ccdd","Type":"ContainerDied","Data":"a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b"} Nov 02 20:36:53.735375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.735193 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx" event={"ID":"0050aa91-905a-4fc3-835a-9c557c04ccdd","Type":"ContainerDied","Data":"0487212f4af70ee37e97ea787684bebd358ff690da4ac45b654955ba92d01197"} Nov 02 20:36:53.735375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.735221 5399 scope.go:122] "RemoveContainer" containerID="a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b" Nov 02 20:36:53.736951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.736927 5399 generic.go:356] "Generic (PLEG): container finished" podID="6677808e-467a-4e97-be61-95b8f53df853" containerID="1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e" exitCode=0 Nov 02 20:36:53.737081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.736959 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" event={"ID":"6677808e-467a-4e97-be61-95b8f53df853","Type":"ContainerDied","Data":"1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e"} Nov 02 20:36:53.737081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.737018 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" event={"ID":"6677808e-467a-4e97-be61-95b8f53df853","Type":"ContainerDied","Data":"685b577f25f6eb082662fb76f9aab6d3e88dd8245d6496b714f2a667363eddcd"} Nov 02 20:36:53.737081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.736970 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m" Nov 02 20:36:53.745428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.745408 5399 scope.go:122] "RemoveContainer" containerID="a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b" Nov 02 20:36:53.745701 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:36:53.745679 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b\": container with ID starting with a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b not found: ID does not exist" containerID="a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b" Nov 02 20:36:53.745780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.745714 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b"} err="failed to get container status \"a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b\": rpc error: code = NotFound desc = could not find container \"a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b\": container with ID starting with a697e38892f0113612e48d28319529e9fc2964b0a06262845879d3500104f08b not found: ID does not exist" Nov 02 20:36:53.745780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.745740 5399 scope.go:122] "RemoveContainer" containerID="1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e" Nov 02 20:36:53.759048 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.759029 5399 scope.go:122] "RemoveContainer" containerID="1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e" Nov 02 20:36:53.759348 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:36:53.759327 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e\": container with ID starting with 1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e not found: ID does not exist" containerID="1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e" Nov 02 20:36:53.759411 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.759358 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e"} err="failed to get container status \"1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e\": rpc error: code = NotFound desc = could not find container \"1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e\": container with ID starting with 1bbf99fc7bc496ee47bb51fb4ae1ac1b32382a5f8e519ded94c55ceedc26fb7e not found: ID does not exist" Nov 02 20:36:53.772885 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.772857 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0050aa91-905a-4fc3-835a-9c557c04ccdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0050aa91-905a-4fc3-835a-9c557c04ccdd-plugins-config-volume\") pod \"0050aa91-905a-4fc3-835a-9c557c04ccdd\" (UID: \"0050aa91-905a-4fc3-835a-9c557c04ccdd\") " Nov 02 20:36:53.772998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.772891 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0050aa91-905a-4fc3-835a-9c557c04ccdd-kube-api-access-zfsv5\" (UniqueName: \"kubernetes.io/projected/0050aa91-905a-4fc3-835a-9c557c04ccdd-kube-api-access-zfsv5\") pod \"0050aa91-905a-4fc3-835a-9c557c04ccdd\" (UID: \"0050aa91-905a-4fc3-835a-9c557c04ccdd\") " Nov 02 20:36:53.773237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.773214 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0050aa91-905a-4fc3-835a-9c557c04ccdd-plugins-config-volume" pod "0050aa91-905a-4fc3-835a-9c557c04ccdd" (UID: "0050aa91-905a-4fc3-835a-9c557c04ccdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:53.773360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.773332 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82xb8\" (UniqueName: \"kubernetes.io/projected/6677808e-467a-4e97-be61-95b8f53df853-kube-api-access-82xb8\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:53.773360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.773346 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0050aa91-905a-4fc3-835a-9c557c04ccdd-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:53.775324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.775304 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0050aa91-905a-4fc3-835a-9c557c04ccdd-kube-api-access-zfsv5" pod "0050aa91-905a-4fc3-835a-9c557c04ccdd" (UID: "0050aa91-905a-4fc3-835a-9c557c04ccdd"). InnerVolumeSpecName "kube-api-access-zfsv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:53.874577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.874472 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfsv5\" (UniqueName: \"kubernetes.io/projected/0050aa91-905a-4fc3-835a-9c557c04ccdd-kube-api-access-zfsv5\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:53.913042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.912990 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m"] Nov 02 20:36:53.915988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:53.915958 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-982b8a-767bc6ddcf-kjf7m"] Nov 02 20:36:54.150108 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:54.150029 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx"] Nov 02 20:36:54.152980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:54.152952 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-982b8a-79fffccf79-2hmtx"] Nov 02 20:36:55.292250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.292209 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq"] Nov 02 20:36:55.293952 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.293924 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e4cf1f20-7f73-4775-b070-1f3ce66c8973" containerName="revoke-keys" Nov 02 20:36:55.294113 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.294100 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cf1f20-7f73-4775-b070-1f3ce66c8973" containerName="revoke-keys" Nov 02 20:36:55.294343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.294327 5399 cpu_manager.go:414] "containerMap: removing container" podUID="0050aa91-905a-4fc3-835a-9c557c04ccdd" containerName="payload-pre-processing" Nov 02 20:36:55.294447 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.294435 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0050aa91-905a-4fc3-835a-9c557c04ccdd" containerName="payload-pre-processing" Nov 02 20:36:55.294579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.294570 5399 cpu_manager.go:414] "containerMap: removing container" podUID="6677808e-467a-4e97-be61-95b8f53df853" containerName="payload-processing" Nov 02 20:36:55.294660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.294649 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6677808e-467a-4e97-be61-95b8f53df853" containerName="payload-processing" Nov 02 20:36:55.294972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.294955 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6677808e-467a-4e97-be61-95b8f53df853" containerName="payload-processing" Nov 02 20:36:55.295084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.295073 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cf1f20-7f73-4775-b070-1f3ce66c8973" containerName="revoke-keys" Nov 02 20:36:55.295185 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.295174 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="0050aa91-905a-4fc3-835a-9c557c04ccdd" containerName="payload-pre-processing" Nov 02 20:36:55.301644 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.301609 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.346682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.346650 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq"] Nov 02 20:36:55.390127 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.390095 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:36:55.390344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.390180 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:36:55.398844 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.398809 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b04f4e30-qw8qq\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.399021 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.398892 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b04f4e30-qw8qq\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.399104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.399071 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fw8\" (UniqueName: \"kubernetes.io/projected/ed0f2396-8f87-4086-98bd-20b5250e157d-kube-api-access-m9fw8\") pod \"maas-api-revoke-sub-b04f4e30-qw8qq\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.500220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.500185 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fw8\" (UniqueName: \"kubernetes.io/projected/ed0f2396-8f87-4086-98bd-20b5250e157d-kube-api-access-m9fw8\") pod \"maas-api-revoke-sub-b04f4e30-qw8qq\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.500434 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.500354 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b04f4e30-qw8qq\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.500434 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.500422 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b04f4e30-qw8qq\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.501179 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.501146 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b04f4e30-qw8qq\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.503143 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.503120 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b04f4e30-qw8qq\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.507795 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.507769 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0050aa91-905a-4fc3-835a-9c557c04ccdd" path="/var/lib/kubelet/pods/0050aa91-905a-4fc3-835a-9c557c04ccdd/volumes" Nov 02 20:36:55.508172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.508158 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6677808e-467a-4e97-be61-95b8f53df853" path="/var/lib/kubelet/pods/6677808e-467a-4e97-be61-95b8f53df853/volumes" Nov 02 20:36:55.557418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.557395 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fw8\" (UniqueName: \"kubernetes.io/projected/ed0f2396-8f87-4086-98bd-20b5250e157d-kube-api-access-m9fw8\") pod \"maas-api-revoke-sub-b04f4e30-qw8qq\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.615396 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.615368 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:55.994891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:55.994860 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq"] Nov 02 20:36:55.996185 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:36:55.996148 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0f2396_8f87_4086_98bd_20b5250e157d.slice/crio-692bf6a9d0269a14aa956f91247eccc9edafc980ab11ca31717865cae256e0be WatchSource:0}: Error finding container 692bf6a9d0269a14aa956f91247eccc9edafc980ab11ca31717865cae256e0be: Status 404 returned error can't find the container with id 692bf6a9d0269a14aa956f91247eccc9edafc980ab11ca31717865cae256e0be Nov 02 20:36:56.773374 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:56.773337 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" event={"ID":"ed0f2396-8f87-4086-98bd-20b5250e157d","Type":"ContainerStarted","Data":"c697e658e41d92158478c076e54c8ad73836367a25fa0f20d90a7dca0079a1ca"} Nov 02 20:36:56.773823 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:56.773396 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" event={"ID":"ed0f2396-8f87-4086-98bd-20b5250e157d","Type":"ContainerStarted","Data":"692bf6a9d0269a14aa956f91247eccc9edafc980ab11ca31717865cae256e0be"} Nov 02 20:36:56.881948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:56.881892 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" podStartSLOduration=1.8818765750000002 podStartE2EDuration="1.881876575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:56.878946117 +0000 UTC m=+2659.489598207" watchObservedRunningTime="2026-11-02 20:36:56.881876575 +0000 UTC m=+2659.492528666" Nov 02 20:36:57.781427 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:57.781395 5399 generic.go:356] "Generic (PLEG): container finished" podID="ed0f2396-8f87-4086-98bd-20b5250e157d" containerID="c697e658e41d92158478c076e54c8ad73836367a25fa0f20d90a7dca0079a1ca" exitCode=0 Nov 02 20:36:57.781843 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:57.781473 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" event={"ID":"ed0f2396-8f87-4086-98bd-20b5250e157d","Type":"ContainerDied","Data":"c697e658e41d92158478c076e54c8ad73836367a25fa0f20d90a7dca0079a1ca"} Nov 02 20:36:59.078755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.078728 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:59.163669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.162616 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0f2396-8f87-4086-98bd-20b5250e157d-kube-api-access-m9fw8\" (UniqueName: \"kubernetes.io/projected/ed0f2396-8f87-4086-98bd-20b5250e157d-kube-api-access-m9fw8\") pod \"ed0f2396-8f87-4086-98bd-20b5250e157d\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " Nov 02 20:36:59.163669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.162725 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-openshift-ca\") pod \"ed0f2396-8f87-4086-98bd-20b5250e157d\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " Nov 02 20:36:59.163669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.162869 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-xks-ca\") pod \"ed0f2396-8f87-4086-98bd-20b5250e157d\" (UID: \"ed0f2396-8f87-4086-98bd-20b5250e157d\") " Nov 02 20:36:59.164033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.164001 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-openshift-ca" pod "ed0f2396-8f87-4086-98bd-20b5250e157d" (UID: "ed0f2396-8f87-4086-98bd-20b5250e157d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:36:59.165785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.165749 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-xks-ca" pod "ed0f2396-8f87-4086-98bd-20b5250e157d" (UID: "ed0f2396-8f87-4086-98bd-20b5250e157d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:36:59.167450 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.167420 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0f2396-8f87-4086-98bd-20b5250e157d-kube-api-access-m9fw8" pod "ed0f2396-8f87-4086-98bd-20b5250e157d" (UID: "ed0f2396-8f87-4086-98bd-20b5250e157d"). InnerVolumeSpecName "kube-api-access-m9fw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:36:59.264675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.264640 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fw8\" (UniqueName: \"kubernetes.io/projected/ed0f2396-8f87-4086-98bd-20b5250e157d-kube-api-access-m9fw8\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:59.264675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.264672 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:59.264675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.264684 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed0f2396-8f87-4086-98bd-20b5250e157d-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:36:59.805798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.805772 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" Nov 02 20:36:59.805952 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.805785 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq" event={"ID":"ed0f2396-8f87-4086-98bd-20b5250e157d","Type":"ContainerDied","Data":"692bf6a9d0269a14aa956f91247eccc9edafc980ab11ca31717865cae256e0be"} Nov 02 20:36:59.805952 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:36:59.805867 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="692bf6a9d0269a14aa956f91247eccc9edafc980ab11ca31717865cae256e0be" Nov 02 20:37:07.892139 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:07.892103 5399 generic.go:356] "Generic (PLEG): container finished" podID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerID="45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19" exitCode=0 Nov 02 20:37:07.892609 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:07.892177 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" event={"ID":"6debae9d-b9fb-40b8-9863-76e329edfdaa","Type":"ContainerDied","Data":"45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19"} Nov 02 20:37:07.892609 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:07.892596 5399 scope.go:122] "RemoveContainer" containerID="45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19" Nov 02 20:37:07.893970 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:07.893945 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:07.894044 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:07.894014 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:07.894082 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:07.894046 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:08.902354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:08.902315 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" event={"ID":"6debae9d-b9fb-40b8-9863-76e329edfdaa","Type":"ContainerStarted","Data":"f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce"} Nov 02 20:37:09.247358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:09.246749 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:37:09.247358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:09.246879 5399 patch_prober.go:29] interesting pod/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:37:09.247358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:09.246921 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:37:10.240308 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:10.239695 5399 patch_prober.go:29] interesting pod/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:37:10.240308 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:10.239757 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:37:11.238759 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:11.238720 5399 patch_prober.go:29] interesting pod/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:37:11.239448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:11.239412 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:37:12.251348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:12.241684 5399 patch_prober.go:29] interesting pod/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:37:12.251348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:12.241765 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:37:13.251304 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:13.240451 5399 patch_prober.go:29] interesting pod/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 02 20:37:13.251304 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:13.240513 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 02 20:37:14.239622 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:14.239585 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:37:14.240138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:14.240121 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:37:14.241061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:14.241040 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:37:18.239311 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:37:18.239258 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:37:20.056384 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:20.056344 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:37:29.043730 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:29.043687 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7"] Nov 02 20:37:29.077913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:29.077881 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r"] Nov 02 20:37:29.090197 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:29.090162 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b52de18-75qh7"] Nov 02 20:37:29.092748 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:29.092724 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-819f6ba3-wv96r"] Nov 02 20:37:29.507149 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:29.507117 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115423d0-4be6-4a3c-832e-72c067cd44a8" path="/var/lib/kubelet/pods/115423d0-4be6-4a3c-832e-72c067cd44a8/volumes" Nov 02 20:37:29.507604 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:29.507588 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a641606-742f-4eb3-9dde-bb2f9ab0646a" path="/var/lib/kubelet/pods/1a641606-742f-4eb3-9dde-bb2f9ab0646a/volumes" Nov 02 20:37:32.693763 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:32.693728 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt"] Nov 02 20:37:32.694251 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:32.693928 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" podUID="0930d956-8e72-4e36-bd77-5ff02a4892a2" containerName="maas-api" containerID="cri-o://1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf" gracePeriod=30 Nov 02 20:37:32.966561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:32.966534 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:37:33.063935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.063885 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-maas-api-tls\") pod \"0930d956-8e72-4e36-bd77-5ff02a4892a2\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " Nov 02 20:37:33.064138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.063949 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0930d956-8e72-4e36-bd77-5ff02a4892a2-kube-api-access-977zd\" (UniqueName: \"kubernetes.io/projected/0930d956-8e72-4e36-bd77-5ff02a4892a2-kube-api-access-977zd\") pod \"0930d956-8e72-4e36-bd77-5ff02a4892a2\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " Nov 02 20:37:33.064138 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.064087 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0930d956-8e72-4e36-bd77-5ff02a4892a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0930d956-8e72-4e36-bd77-5ff02a4892a2-openshift-service-ca\") pod \"0930d956-8e72-4e36-bd77-5ff02a4892a2\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " Nov 02 20:37:33.064268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.064147 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-metrics-certs\") pod \"0930d956-8e72-4e36-bd77-5ff02a4892a2\" (UID: \"0930d956-8e72-4e36-bd77-5ff02a4892a2\") " Nov 02 20:37:33.072096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.071968 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-metrics-certs" pod "0930d956-8e72-4e36-bd77-5ff02a4892a2" (UID: "0930d956-8e72-4e36-bd77-5ff02a4892a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:33.074721 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.074558 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-maas-api-tls" pod "0930d956-8e72-4e36-bd77-5ff02a4892a2" (UID: "0930d956-8e72-4e36-bd77-5ff02a4892a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:33.076703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.076644 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0930d956-8e72-4e36-bd77-5ff02a4892a2-openshift-service-ca" pod "0930d956-8e72-4e36-bd77-5ff02a4892a2" (UID: "0930d956-8e72-4e36-bd77-5ff02a4892a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:33.089487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.084404 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0930d956-8e72-4e36-bd77-5ff02a4892a2-kube-api-access-977zd" pod "0930d956-8e72-4e36-bd77-5ff02a4892a2" (UID: "0930d956-8e72-4e36-bd77-5ff02a4892a2"). InnerVolumeSpecName "kube-api-access-977zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:33.111831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.111803 5399 generic.go:356] "Generic (PLEG): container finished" podID="0930d956-8e72-4e36-bd77-5ff02a4892a2" containerID="1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf" exitCode=0 Nov 02 20:37:33.111960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.111846 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" Nov 02 20:37:33.111960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.111890 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" event={"ID":"0930d956-8e72-4e36-bd77-5ff02a4892a2","Type":"ContainerDied","Data":"1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf"} Nov 02 20:37:33.111960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.111947 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt" event={"ID":"0930d956-8e72-4e36-bd77-5ff02a4892a2","Type":"ContainerDied","Data":"e51b02c561323f0eed61469db8ad7c2a372bb817a285dd8d73bea55b1fbf6cba"} Nov 02 20:37:33.112075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.111970 5399 scope.go:122] "RemoveContainer" containerID="1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf" Nov 02 20:37:33.122227 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.122208 5399 scope.go:122] "RemoveContainer" containerID="1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf" Nov 02 20:37:33.122535 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:37:33.122512 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf\": container with ID starting with 1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf not found: ID does not exist" containerID="1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf" Nov 02 20:37:33.122620 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.122547 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf"} err="failed to get container status \"1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf\": rpc error: code = NotFound desc = could not find container \"1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf\": container with ID starting with 1041dda10414adea43e72cc6d8bfad0a2558d459b62c7c0e67e6766e7dafeabf not found: ID does not exist" Nov 02 20:37:33.165892 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.165861 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0930d956-8e72-4e36-bd77-5ff02a4892a2-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:33.166054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.165911 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:33.166054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.165921 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0930d956-8e72-4e36-bd77-5ff02a4892a2-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:33.166054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.165932 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-977zd\" (UniqueName: \"kubernetes.io/projected/0930d956-8e72-4e36-bd77-5ff02a4892a2-kube-api-access-977zd\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:33.264881 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.264804 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt"] Nov 02 20:37:33.274410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.274379 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2052e3-5fdc798599-pxtkt"] Nov 02 20:37:33.508867 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:33.508837 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0930d956-8e72-4e36-bd77-5ff02a4892a2" path="/var/lib/kubelet/pods/0930d956-8e72-4e36-bd77-5ff02a4892a2/volumes" Nov 02 20:37:34.723087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:34.723053 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz"] Nov 02 20:37:34.760447 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:34.760415 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2052e3-x2txz"] Nov 02 20:37:35.507410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:35.507377 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13" path="/var/lib/kubelet/pods/d3cc6b53-fcfa-4e44-a4a1-e6f09de5ad13/volumes" Nov 02 20:37:38.943495 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:38.943459 5399 scope.go:122] "RemoveContainer" containerID="01eeab4b220ff859c534e0c89559d60b845772769d2d2babda54234a8f4e3747" Nov 02 20:37:38.953441 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:38.953411 5399 scope.go:122] "RemoveContainer" containerID="a54913674b2e6d540e541deebb5319e9044e9bd05c362aedb3ed0c45dcac8f72" Nov 02 20:37:38.962420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:38.962399 5399 scope.go:122] "RemoveContainer" containerID="c0c01b847ff34c0d667f3ec3f13c6318cab0559e0bc5f32884da831826cbf686" Nov 02 20:37:48.242732 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:37:48.242638 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:37:49.668768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:49.668733 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5"] Nov 02 20:37:49.715028 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:49.714993 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-982b8a-z5cz5"] Nov 02 20:37:51.507540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:37:51.507508 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea2fe64-b1a3-44aa-8f92-d879baa2b40e" path="/var/lib/kubelet/pods/3ea2fe64-b1a3-44aa-8f92-d879baa2b40e/volumes" Nov 02 20:38:11.936712 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:11.936673 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:38:18.237385 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:38:18.237347 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:38:39.015017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:39.014982 5399 scope.go:122] "RemoveContainer" containerID="ed4d563126e00dc9713c8a7a24b0d3951382414c7744b619bebd044a5bf93ecb" Nov 02 20:38:39.024856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:39.024835 5399 scope.go:122] "RemoveContainer" containerID="c35c3b8ff720b3373eb42b46fac843dcf55737595f53629b4bdb7e640e614016" Nov 02 20:38:39.034037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:39.034014 5399 scope.go:122] "RemoveContainer" containerID="4ef517afe6792a6200d45fb057fb217afa9a64e6dd528491db4ca46dda9c5cd4" Nov 02 20:38:39.042459 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:39.042436 5399 scope.go:122] "RemoveContainer" containerID="4d22da0202cfdbb357e4c99e93efbd95b9e49a6ae1593b06e5d1ae93e31882d6" Nov 02 20:38:39.051501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:39.051479 5399 scope.go:122] "RemoveContainer" containerID="5fe8148d41f6af5715451f10f83b151af7ed8d2b24f8d2d0ea33847479dacae9" Nov 02 20:38:46.171200 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:46.171166 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:38:48.237129 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:38:48.237092 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:38:51.836041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:51.836005 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5"] Nov 02 20:38:51.836599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:51.836241 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" podUID="a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" containerName="discovery" containerID="cri-o://e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f" gracePeriod=30 Nov 02 20:38:52.104960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.104927 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:38:52.158031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.158002 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-dns-cert\") pod \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " Nov 02 20:38:52.158201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.158087 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-kubeconfig\") pod \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " Nov 02 20:38:52.158201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.158109 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-cacerts\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-cacerts\") pod \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " Nov 02 20:38:52.158201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.158130 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-token\" (UniqueName: \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-token\") pod \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " Nov 02 20:38:52.158201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.158194 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-ca-configmap\") pod \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " Nov 02 20:38:52.158467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.158246 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-kube-api-access-pvfxs\" (UniqueName: \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-kube-api-access-pvfxs\") pod \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " Nov 02 20:38:52.158467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.158422 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-local-certs\") pod \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\" (UID: \"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f\") " Nov 02 20:38:52.158639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.158594 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-ca-configmap" pod "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" (UID: "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:52.158912 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.158889 5399 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-ca-configmap\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:52.160536 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.160508 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-cacerts" pod "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" (UID: "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:52.160802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.160775 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-token" pod "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" (UID: "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:52.160887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.160841 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-local-certs" pod "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" (UID: "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:38:52.160887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.160844 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-kubeconfig" pod "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" (UID: "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:52.161062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.161044 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-kube-api-access-pvfxs" pod "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" (UID: "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f"). InnerVolumeSpecName "kube-api-access-pvfxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:52.161123 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.161043 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-dns-cert" pod "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" (UID: "a7ac9c5f-54f3-4ba8-911c-ebe5d560516f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:52.260588 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.260556 5399 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-local-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:52.260588 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.260587 5399 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-csr-dns-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:52.260588 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.260596 5399 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-kubeconfig\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:52.260831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.260605 5399 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-cacerts\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:52.260831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.260616 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:52.260831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.260625 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvfxs\" (UniqueName: \"kubernetes.io/projected/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f-kube-api-access-pvfxs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:52.669781 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.669743 5399 generic.go:356] "Generic (PLEG): container finished" podID="a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" containerID="e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f" exitCode=0 Nov 02 20:38:52.669998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.669829 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" Nov 02 20:38:52.669998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.669825 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" event={"ID":"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f","Type":"ContainerDied","Data":"e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f"} Nov 02 20:38:52.669998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.669938 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5" event={"ID":"a7ac9c5f-54f3-4ba8-911c-ebe5d560516f","Type":"ContainerDied","Data":"26c4d689f7166d59d0494ea188a6aaece863dc0e897feedb772d63ecc456432b"} Nov 02 20:38:52.669998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.669959 5399 scope.go:122] "RemoveContainer" containerID="e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f" Nov 02 20:38:52.684324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.684305 5399 scope.go:122] "RemoveContainer" containerID="e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f" Nov 02 20:38:52.684610 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:38:52.684592 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f\": container with ID starting with e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f not found: ID does not exist" containerID="e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f" Nov 02 20:38:52.684692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.684618 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f"} err="failed to get container status \"e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f\": rpc error: code = NotFound desc = could not find container \"e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f\": container with ID starting with e56b49a8edb47a633a5b68826930b5460a5968a2f84595d59d010cb0d58d1a9f not found: ID does not exist" Nov 02 20:38:52.824306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.824254 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5"] Nov 02 20:38:52.871060 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:52.871022 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9rlz5"] Nov 02 20:38:53.507501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:38:53.507471 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" path="/var/lib/kubelet/pods/a7ac9c5f-54f3-4ba8-911c-ebe5d560516f/volumes" Nov 02 20:39:18.237078 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:39:18.236983 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:39:22.821488 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:22.821450 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:39:30.403890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.403855 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l"] Nov 02 20:39:30.404408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.404393 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ed0f2396-8f87-4086-98bd-20b5250e157d" containerName="revoke-keys" Nov 02 20:39:30.404461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.404408 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0f2396-8f87-4086-98bd-20b5250e157d" containerName="revoke-keys" Nov 02 20:39:30.404461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.404430 5399 cpu_manager.go:414] "containerMap: removing container" podUID="0930d956-8e72-4e36-bd77-5ff02a4892a2" containerName="maas-api" Nov 02 20:39:30.404461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.404435 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0930d956-8e72-4e36-bd77-5ff02a4892a2" containerName="maas-api" Nov 02 20:39:30.404461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.404446 5399 cpu_manager.go:414] "containerMap: removing container" podUID="a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" containerName="discovery" Nov 02 20:39:30.404461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.404450 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" containerName="discovery" Nov 02 20:39:30.404619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.404545 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0f2396-8f87-4086-98bd-20b5250e157d" containerName="revoke-keys" Nov 02 20:39:30.404619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.404555 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="0930d956-8e72-4e36-bd77-5ff02a4892a2" containerName="maas-api" Nov 02 20:39:30.404619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.404562 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ac9c5f-54f3-4ba8-911c-ebe5d560516f" containerName="discovery" Nov 02 20:39:30.408036 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.408012 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:30.459802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.459773 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l"] Nov 02 20:39:30.491113 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.491080 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc001fc9-pv72l\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:30.491291 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.491130 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc001fc9-pv72l\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:30.491291 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.491203 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g8vf\" (UniqueName: \"kubernetes.io/projected/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-kube-api-access-5g8vf\") pod \"maas-api-revoke-sub-fc001fc9-pv72l\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:30.503041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.503013 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:39:30.503186 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.503070 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:39:30.592363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.592332 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc001fc9-pv72l\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:30.592571 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.592376 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc001fc9-pv72l\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:30.592571 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.592397 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g8vf\" (UniqueName: \"kubernetes.io/projected/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-kube-api-access-5g8vf\") pod \"maas-api-revoke-sub-fc001fc9-pv72l\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:30.592981 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.592958 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc001fc9-pv72l\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:30.594751 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.594726 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc001fc9-pv72l\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:30.645782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.645749 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g8vf\" (UniqueName: \"kubernetes.io/projected/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-kube-api-access-5g8vf\") pod \"maas-api-revoke-sub-fc001fc9-pv72l\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:30.722041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:30.721963 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:31.087428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:31.087399 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l"] Nov 02 20:39:31.088693 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:39:31.088670 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4347a86f_c4c4_45cf_b812_4d1e79e4d0bc.slice/crio-ee21f6daf8fbeba198e4ce79850d7db55680ca6171fa6e8091a1bac62d2f43b7 WatchSource:0}: Error finding container ee21f6daf8fbeba198e4ce79850d7db55680ca6171fa6e8091a1bac62d2f43b7: Status 404 returned error can't find the container with id ee21f6daf8fbeba198e4ce79850d7db55680ca6171fa6e8091a1bac62d2f43b7 Nov 02 20:39:31.941821 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:31.941784 5399 generic.go:356] "Generic (PLEG): container finished" podID="4347a86f-c4c4-45cf-b812-4d1e79e4d0bc" containerID="28784316d65c91b938963531ddf55d3afb47cbd1cfbdab8a2186e8627b5e0843" exitCode=0 Nov 02 20:39:31.942185 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:31.941858 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" event={"ID":"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc","Type":"ContainerDied","Data":"28784316d65c91b938963531ddf55d3afb47cbd1cfbdab8a2186e8627b5e0843"} Nov 02 20:39:31.942185 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:31.941907 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" event={"ID":"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc","Type":"ContainerStarted","Data":"ee21f6daf8fbeba198e4ce79850d7db55680ca6171fa6e8091a1bac62d2f43b7"} Nov 02 20:39:33.095420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.095269 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:33.223914 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.223829 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-kube-api-access-5g8vf\" (UniqueName: \"kubernetes.io/projected/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-kube-api-access-5g8vf\") pod \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " Nov 02 20:39:33.224075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.223933 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-openshift-ca\") pod \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " Nov 02 20:39:33.224146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.224103 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-xks-ca\") pod \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\" (UID: \"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc\") " Nov 02 20:39:33.224221 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.224204 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-openshift-ca" pod "4347a86f-c4c4-45cf-b812-4d1e79e4d0bc" (UID: "4347a86f-c4c4-45cf-b812-4d1e79e4d0bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:33.224596 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.224569 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:33.226199 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.226173 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-kube-api-access-5g8vf" pod "4347a86f-c4c4-45cf-b812-4d1e79e4d0bc" (UID: "4347a86f-c4c4-45cf-b812-4d1e79e4d0bc"). InnerVolumeSpecName "kube-api-access-5g8vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:33.226599 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.226579 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-xks-ca" pod "4347a86f-c4c4-45cf-b812-4d1e79e4d0bc" (UID: "4347a86f-c4c4-45cf-b812-4d1e79e4d0bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:39:33.325760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.325722 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g8vf\" (UniqueName: \"kubernetes.io/projected/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-kube-api-access-5g8vf\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:33.325760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.325758 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:33.957174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.957141 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" Nov 02 20:39:33.957174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.957167 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l" event={"ID":"4347a86f-c4c4-45cf-b812-4d1e79e4d0bc","Type":"ContainerDied","Data":"ee21f6daf8fbeba198e4ce79850d7db55680ca6171fa6e8091a1bac62d2f43b7"} Nov 02 20:39:33.957426 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:33.957219 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee21f6daf8fbeba198e4ce79850d7db55680ca6171fa6e8091a1bac62d2f43b7" Nov 02 20:39:39.092895 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:39.092856 5399 scope.go:122] "RemoveContainer" containerID="f69fa82bfa0e7590b42ad742ec4532cc0ad90bb4032e611e5f6d7e6dfefa6f51" Nov 02 20:39:39.102483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:39.102460 5399 scope.go:122] "RemoveContainer" containerID="8368aaa891deeff6641a40351e3631345f3457a013f90fd3c8851aea73b40b32" Nov 02 20:39:39.111096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:39.111078 5399 scope.go:122] "RemoveContainer" containerID="aa485c3d91f36035d120346a219ac6c6a20ed5b0ed4ebdecda98fe38529224c6" Nov 02 20:39:43.249555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.249512 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc"] Nov 02 20:39:43.250247 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.250224 5399 cpu_manager.go:414] "containerMap: removing container" podUID="4347a86f-c4c4-45cf-b812-4d1e79e4d0bc" containerName="revoke-keys" Nov 02 20:39:43.250381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.250248 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4347a86f-c4c4-45cf-b812-4d1e79e4d0bc" containerName="revoke-keys" Nov 02 20:39:43.250452 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.250399 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="4347a86f-c4c4-45cf-b812-4d1e79e4d0bc" containerName="revoke-keys" Nov 02 20:39:43.255863 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.255830 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.307174 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.307140 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc"] Nov 02 20:39:43.307527 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.307507 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:39:43.330495 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.330458 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a206ebf6-g2bzc\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.330706 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.330507 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgddk\" (UniqueName: \"kubernetes.io/projected/555b5b76-927c-4248-8a59-5a4cc6f68dcb-kube-api-access-bgddk\") pod \"maas-api-revoke-sub-a206ebf6-g2bzc\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.330706 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.330620 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a206ebf6-g2bzc\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.349397 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.349364 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:39:43.431350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.431304 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a206ebf6-g2bzc\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.431350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.431358 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgddk\" (UniqueName: \"kubernetes.io/projected/555b5b76-927c-4248-8a59-5a4cc6f68dcb-kube-api-access-bgddk\") pod \"maas-api-revoke-sub-a206ebf6-g2bzc\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.431624 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.431410 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a206ebf6-g2bzc\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.432399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.432368 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a206ebf6-g2bzc\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.434912 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.434878 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a206ebf6-g2bzc\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.484639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.484605 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgddk\" (UniqueName: \"kubernetes.io/projected/555b5b76-927c-4248-8a59-5a4cc6f68dcb-kube-api-access-bgddk\") pod \"maas-api-revoke-sub-a206ebf6-g2bzc\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.571500 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.571469 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:43.946149 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:43.943648 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc"] Nov 02 20:39:43.948227 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:39:43.948198 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555b5b76_927c_4248_8a59_5a4cc6f68dcb.slice/crio-9e712d79789265f1b09e59b29ccf181566db662c4f6b8e35e99a0f09e0fbe7b9 WatchSource:0}: Error finding container 9e712d79789265f1b09e59b29ccf181566db662c4f6b8e35e99a0f09e0fbe7b9: Status 404 returned error can't find the container with id 9e712d79789265f1b09e59b29ccf181566db662c4f6b8e35e99a0f09e0fbe7b9 Nov 02 20:39:44.018087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:44.018050 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" event={"ID":"555b5b76-927c-4248-8a59-5a4cc6f68dcb","Type":"ContainerStarted","Data":"9e712d79789265f1b09e59b29ccf181566db662c4f6b8e35e99a0f09e0fbe7b9"} Nov 02 20:39:45.025673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:45.025637 5399 generic.go:356] "Generic (PLEG): container finished" podID="555b5b76-927c-4248-8a59-5a4cc6f68dcb" containerID="a1520abf5427da0839c5354d2389c2e454c0c2189823545054b238a60e197ca8" exitCode=0 Nov 02 20:39:45.026069 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:45.025719 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" event={"ID":"555b5b76-927c-4248-8a59-5a4cc6f68dcb","Type":"ContainerDied","Data":"a1520abf5427da0839c5354d2389c2e454c0c2189823545054b238a60e197ca8"} Nov 02 20:39:46.181633 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:46.181608 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:46.263569 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:46.263532 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-xks-ca\") pod \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " Nov 02 20:39:46.263762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:46.263585 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-openshift-ca\") pod \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " Nov 02 20:39:46.263762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:46.263614 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555b5b76-927c-4248-8a59-5a4cc6f68dcb-kube-api-access-bgddk\" (UniqueName: \"kubernetes.io/projected/555b5b76-927c-4248-8a59-5a4cc6f68dcb-kube-api-access-bgddk\") pod \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\" (UID: \"555b5b76-927c-4248-8a59-5a4cc6f68dcb\") " Nov 02 20:39:46.264008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:46.263984 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-openshift-ca" pod "555b5b76-927c-4248-8a59-5a4cc6f68dcb" (UID: "555b5b76-927c-4248-8a59-5a4cc6f68dcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:46.265680 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:46.265646 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-xks-ca" pod "555b5b76-927c-4248-8a59-5a4cc6f68dcb" (UID: "555b5b76-927c-4248-8a59-5a4cc6f68dcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:39:46.265792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:46.265696 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555b5b76-927c-4248-8a59-5a4cc6f68dcb-kube-api-access-bgddk" pod "555b5b76-927c-4248-8a59-5a4cc6f68dcb" (UID: "555b5b76-927c-4248-8a59-5a4cc6f68dcb"). InnerVolumeSpecName "kube-api-access-bgddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:46.364850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:46.364823 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:46.364850 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:46.364847 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555b5b76-927c-4248-8a59-5a4cc6f68dcb-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:46.365046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:46.364858 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgddk\" (UniqueName: \"kubernetes.io/projected/555b5b76-927c-4248-8a59-5a4cc6f68dcb-kube-api-access-bgddk\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:47.039124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:47.039094 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" Nov 02 20:39:47.039313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:47.039125 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc" event={"ID":"555b5b76-927c-4248-8a59-5a4cc6f68dcb","Type":"ContainerDied","Data":"9e712d79789265f1b09e59b29ccf181566db662c4f6b8e35e99a0f09e0fbe7b9"} Nov 02 20:39:47.039313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:47.039172 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e712d79789265f1b09e59b29ccf181566db662c4f6b8e35e99a0f09e0fbe7b9" Nov 02 20:39:48.238158 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:39:48.238115 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:39:49.044792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:49.044762 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t"] Nov 02 20:39:49.091774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:49.091739 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa60f03-zwt5t"] Nov 02 20:39:49.094078 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:49.094064 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d"] Nov 02 20:39:49.096128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:49.096109 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fed7f16-msz6d"] Nov 02 20:39:49.507666 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:49.507635 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e" path="/var/lib/kubelet/pods/b82e5ffc-e0b0-4507-9507-ff96f1e3fd7e/volumes" Nov 02 20:39:49.508061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:49.508051 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b1bec4-3a9e-43dd-b96f-f36af89dbc42" path="/var/lib/kubelet/pods/c5b1bec4-3a9e-43dd-b96f-f36af89dbc42/volumes" Nov 02 20:39:57.962189 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:39:57.962150 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:40:05.320720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.320685 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6"] Nov 02 20:40:05.321239 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.321224 5399 cpu_manager.go:414] "containerMap: removing container" podUID="555b5b76-927c-4248-8a59-5a4cc6f68dcb" containerName="revoke-keys" Nov 02 20:40:05.321337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.321240 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555b5b76-927c-4248-8a59-5a4cc6f68dcb" containerName="revoke-keys" Nov 02 20:40:05.321387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.321339 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="555b5b76-927c-4248-8a59-5a4cc6f68dcb" containerName="revoke-keys" Nov 02 20:40:05.324545 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.324528 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:05.373694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.373662 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:40:05.373886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.373822 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:40:05.377075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.377049 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6"] Nov 02 20:40:05.482694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.482656 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97v9\" (UniqueName: \"kubernetes.io/projected/4479f1cd-e6a6-4165-92a7-8856cb01d449-kube-api-access-t97v9\") pod \"maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:05.482880 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.482809 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:05.482880 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.482869 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:05.583734 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.583634 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:05.583734 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.583709 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97v9\" (UniqueName: \"kubernetes.io/projected/4479f1cd-e6a6-4165-92a7-8856cb01d449-kube-api-access-t97v9\") pod \"maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:05.583941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.583860 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:05.584384 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.584350 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:05.595845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.595810 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:05.637073 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.637037 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97v9\" (UniqueName: \"kubernetes.io/projected/4479f1cd-e6a6-4165-92a7-8856cb01d449-kube-api-access-t97v9\") pod \"maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:05.638677 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:05.638657 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:06.004176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:06.004147 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6"] Nov 02 20:40:06.005840 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:40:06.005807 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4479f1cd_e6a6_4165_92a7_8856cb01d449.slice/crio-2f7408eea0ef2debfe2078aed1105812baf05bc226525a11603e62618c6d4db9 WatchSource:0}: Error finding container 2f7408eea0ef2debfe2078aed1105812baf05bc226525a11603e62618c6d4db9: Status 404 returned error can't find the container with id 2f7408eea0ef2debfe2078aed1105812baf05bc226525a11603e62618c6d4db9 Nov 02 20:40:06.160628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:06.160590 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" event={"ID":"4479f1cd-e6a6-4165-92a7-8856cb01d449","Type":"ContainerStarted","Data":"2f7408eea0ef2debfe2078aed1105812baf05bc226525a11603e62618c6d4db9"} Nov 02 20:40:07.168469 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:07.168432 5399 generic.go:356] "Generic (PLEG): container finished" podID="4479f1cd-e6a6-4165-92a7-8856cb01d449" containerID="25b6b192c71130a2b067698f7d305e1ee9a1e14ac533a7e12a7df0e624f07213" exitCode=0 Nov 02 20:40:07.168898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:07.168515 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" event={"ID":"4479f1cd-e6a6-4165-92a7-8856cb01d449","Type":"ContainerDied","Data":"25b6b192c71130a2b067698f7d305e1ee9a1e14ac533a7e12a7df0e624f07213"} Nov 02 20:40:08.319655 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:08.319628 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:08.418032 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:08.417996 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-xks-ca\") pod \"4479f1cd-e6a6-4165-92a7-8856cb01d449\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " Nov 02 20:40:08.418210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:08.418067 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-openshift-ca\") pod \"4479f1cd-e6a6-4165-92a7-8856cb01d449\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " Nov 02 20:40:08.418210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:08.418171 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4479f1cd-e6a6-4165-92a7-8856cb01d449-kube-api-access-t97v9\" (UniqueName: \"kubernetes.io/projected/4479f1cd-e6a6-4165-92a7-8856cb01d449-kube-api-access-t97v9\") pod \"4479f1cd-e6a6-4165-92a7-8856cb01d449\" (UID: \"4479f1cd-e6a6-4165-92a7-8856cb01d449\") " Nov 02 20:40:08.418479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:08.418454 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-openshift-ca" pod "4479f1cd-e6a6-4165-92a7-8856cb01d449" (UID: "4479f1cd-e6a6-4165-92a7-8856cb01d449"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:40:08.418716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:08.418700 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:08.420311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:08.420270 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-xks-ca" pod "4479f1cd-e6a6-4165-92a7-8856cb01d449" (UID: "4479f1cd-e6a6-4165-92a7-8856cb01d449"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:40:08.420408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:08.420389 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4479f1cd-e6a6-4165-92a7-8856cb01d449-kube-api-access-t97v9" pod "4479f1cd-e6a6-4165-92a7-8856cb01d449" (UID: "4479f1cd-e6a6-4165-92a7-8856cb01d449"). InnerVolumeSpecName "kube-api-access-t97v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:08.519971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:08.519936 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4479f1cd-e6a6-4165-92a7-8856cb01d449-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:08.519971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:08.519965 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t97v9\" (UniqueName: \"kubernetes.io/projected/4479f1cd-e6a6-4165-92a7-8856cb01d449-kube-api-access-t97v9\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:09.183510 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:09.183475 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" event={"ID":"4479f1cd-e6a6-4165-92a7-8856cb01d449","Type":"ContainerDied","Data":"2f7408eea0ef2debfe2078aed1105812baf05bc226525a11603e62618c6d4db9"} Nov 02 20:40:09.183510 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:09.183505 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6" Nov 02 20:40:09.183802 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:09.183524 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f7408eea0ef2debfe2078aed1105812baf05bc226525a11603e62618c6d4db9" Nov 02 20:40:18.239444 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:40:18.239398 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:40:39.150245 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:39.150208 5399 scope.go:122] "RemoveContainer" containerID="de7c2069e200cabef9de8da723dafa9fd34e4e2d7312e012b1c2e0a08d9d2f90" Nov 02 20:40:39.307130 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:39.307095 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:40:48.237904 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:40:48.237820 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:40:50.042906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.042870 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd"] Nov 02 20:40:50.043810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.043147 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" podUID="eee30a99-5158-4da4-a8a6-95c9fd155810" containerName="maas-api" containerID="cri-o://7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6" gracePeriod=30 Nov 02 20:40:50.317567 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.317540 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:40:50.404824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.403754 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee30a99-5158-4da4-a8a6-95c9fd155810-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee30a99-5158-4da4-a8a6-95c9fd155810-openshift-service-ca\") pod \"eee30a99-5158-4da4-a8a6-95c9fd155810\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " Nov 02 20:40:50.404824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.403887 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee30a99-5158-4da4-a8a6-95c9fd155810-kube-api-access-rgdd9\" (UniqueName: \"kubernetes.io/projected/eee30a99-5158-4da4-a8a6-95c9fd155810-kube-api-access-rgdd9\") pod \"eee30a99-5158-4da4-a8a6-95c9fd155810\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " Nov 02 20:40:50.404824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.403944 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-metrics-certs\") pod \"eee30a99-5158-4da4-a8a6-95c9fd155810\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " Nov 02 20:40:50.404824 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.404000 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-maas-api-tls\") pod \"eee30a99-5158-4da4-a8a6-95c9fd155810\" (UID: \"eee30a99-5158-4da4-a8a6-95c9fd155810\") " Nov 02 20:40:50.405677 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.405290 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee30a99-5158-4da4-a8a6-95c9fd155810-openshift-service-ca" pod "eee30a99-5158-4da4-a8a6-95c9fd155810" (UID: "eee30a99-5158-4da4-a8a6-95c9fd155810"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:40:50.412909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.412855 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-maas-api-tls" pod "eee30a99-5158-4da4-a8a6-95c9fd155810" (UID: "eee30a99-5158-4da4-a8a6-95c9fd155810"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:40:50.424381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.424326 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-metrics-certs" pod "eee30a99-5158-4da4-a8a6-95c9fd155810" (UID: "eee30a99-5158-4da4-a8a6-95c9fd155810"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:40:50.433320 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.433261 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee30a99-5158-4da4-a8a6-95c9fd155810-kube-api-access-rgdd9" pod "eee30a99-5158-4da4-a8a6-95c9fd155810" (UID: "eee30a99-5158-4da4-a8a6-95c9fd155810"). InnerVolumeSpecName "kube-api-access-rgdd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:50.455022 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.454987 5399 generic.go:356] "Generic (PLEG): container finished" podID="eee30a99-5158-4da4-a8a6-95c9fd155810" containerID="7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6" exitCode=0 Nov 02 20:40:50.455193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.455035 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" Nov 02 20:40:50.455193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.455067 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" event={"ID":"eee30a99-5158-4da4-a8a6-95c9fd155810","Type":"ContainerDied","Data":"7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6"} Nov 02 20:40:50.455193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.455114 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd" event={"ID":"eee30a99-5158-4da4-a8a6-95c9fd155810","Type":"ContainerDied","Data":"418472c442100cc088b2cb40a8e1e86b1d445b0f0aaba9d178e2d07ab7386445"} Nov 02 20:40:50.455193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.455135 5399 scope.go:122] "RemoveContainer" containerID="7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6" Nov 02 20:40:50.468268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.468248 5399 scope.go:122] "RemoveContainer" containerID="7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6" Nov 02 20:40:50.468634 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:40:50.468615 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6\": container with ID starting with 7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6 not found: ID does not exist" containerID="7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6" Nov 02 20:40:50.468693 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.468644 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6"} err="failed to get container status \"7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6\": rpc error: code = NotFound desc = could not find container \"7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6\": container with ID starting with 7c7ed95309583bb3144e6827f07ee700b4845852096935a77b806a48af58b9a6 not found: ID does not exist" Nov 02 20:40:50.506335 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.506260 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee30a99-5158-4da4-a8a6-95c9fd155810-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:50.506335 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.506332 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgdd9\" (UniqueName: \"kubernetes.io/projected/eee30a99-5158-4da4-a8a6-95c9fd155810-kube-api-access-rgdd9\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:50.506537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.506347 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:50.506537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.506359 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eee30a99-5158-4da4-a8a6-95c9fd155810-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:50.607638 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.607601 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd"] Nov 02 20:40:50.610186 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:50.610161 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65f28f-6d95969d6c-p85bd"] Nov 02 20:40:51.508814 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:51.508773 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee30a99-5158-4da4-a8a6-95c9fd155810" path="/var/lib/kubelet/pods/eee30a99-5158-4da4-a8a6-95c9fd155810/volumes" Nov 02 20:40:52.170162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:52.170126 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6"] Nov 02 20:40:52.216601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:52.216565 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-65f28f-ls7w6"] Nov 02 20:40:53.507244 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:53.507213 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4479f1cd-e6a6-4165-92a7-8856cb01d449" path="/var/lib/kubelet/pods/4479f1cd-e6a6-4165-92a7-8856cb01d449/volumes" Nov 02 20:40:59.799005 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.798972 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc"] Nov 02 20:40:59.799501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.799486 5399 cpu_manager.go:414] "containerMap: removing container" podUID="eee30a99-5158-4da4-a8a6-95c9fd155810" containerName="maas-api" Nov 02 20:40:59.799547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.799502 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee30a99-5158-4da4-a8a6-95c9fd155810" containerName="maas-api" Nov 02 20:40:59.799547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.799526 5399 cpu_manager.go:414] "containerMap: removing container" podUID="4479f1cd-e6a6-4165-92a7-8856cb01d449" containerName="revoke-keys" Nov 02 20:40:59.799547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.799532 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4479f1cd-e6a6-4165-92a7-8856cb01d449" containerName="revoke-keys" Nov 02 20:40:59.799645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.799597 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="4479f1cd-e6a6-4165-92a7-8856cb01d449" containerName="revoke-keys" Nov 02 20:40:59.799645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.799613 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee30a99-5158-4da4-a8a6-95c9fd155810" containerName="maas-api" Nov 02 20:40:59.804343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.804318 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:40:59.855678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.855640 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc"] Nov 02 20:40:59.871544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.871509 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:40:59.898458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.898423 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:40:59.898458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.898432 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:40:59.898458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.898429 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:40:59.915871 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.915840 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:40:59.916034 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.915897 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpjz\" (UniqueName: \"kubernetes.io/projected/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-kube-api-access-dkpjz\") pod \"maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:40:59.916034 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:40:59.915946 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:41:00.017124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:00.017085 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:41:00.017336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:00.017238 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:41:00.017336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:00.017269 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpjz\" (UniqueName: \"kubernetes.io/projected/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-kube-api-access-dkpjz\") pod \"maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:41:00.017779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:00.017756 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:41:00.019634 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:00.019612 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:41:00.070513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:00.070483 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpjz\" (UniqueName: \"kubernetes.io/projected/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-kube-api-access-dkpjz\") pod \"maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:41:00.118196 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:00.118153 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:41:00.488094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:00.488065 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc"] Nov 02 20:41:00.489916 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:41:00.489872 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c8f2f5_3602_4e87_b478_bba9e1bcd355.slice/crio-46bc6ba11212207591dd9b3ad372d9363b4141f26eee4f60a6fe8cfacd29eb02 WatchSource:0}: Error finding container 46bc6ba11212207591dd9b3ad372d9363b4141f26eee4f60a6fe8cfacd29eb02: Status 404 returned error can't find the container with id 46bc6ba11212207591dd9b3ad372d9363b4141f26eee4f60a6fe8cfacd29eb02 Nov 02 20:41:00.522971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:00.522934 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" event={"ID":"d1c8f2f5-3602-4e87-b478-bba9e1bcd355","Type":"ContainerStarted","Data":"46bc6ba11212207591dd9b3ad372d9363b4141f26eee4f60a6fe8cfacd29eb02"} Nov 02 20:41:01.531490 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:01.531449 5399 generic.go:356] "Generic (PLEG): container finished" podID="d1c8f2f5-3602-4e87-b478-bba9e1bcd355" containerID="7dfbdd1f653c19b277be1182f5f4ad3cd9898f7c6b468239a9126e315e26b43a" exitCode=0 Nov 02 20:41:01.531911 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:01.531531 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" event={"ID":"d1c8f2f5-3602-4e87-b478-bba9e1bcd355","Type":"ContainerDied","Data":"7dfbdd1f653c19b277be1182f5f4ad3cd9898f7c6b468239a9126e315e26b43a"} Nov 02 20:41:02.680500 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:02.680462 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:41:02.744562 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:02.744526 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-xks-ca\") pod \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " Nov 02 20:41:02.744743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:02.744624 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-kube-api-access-dkpjz\" (UniqueName: \"kubernetes.io/projected/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-kube-api-access-dkpjz\") pod \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " Nov 02 20:41:02.746607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:02.746579 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-xks-ca" pod "d1c8f2f5-3602-4e87-b478-bba9e1bcd355" (UID: "d1c8f2f5-3602-4e87-b478-bba9e1bcd355"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:41:02.746796 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:02.746769 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-kube-api-access-dkpjz" pod "d1c8f2f5-3602-4e87-b478-bba9e1bcd355" (UID: "d1c8f2f5-3602-4e87-b478-bba9e1bcd355"). InnerVolumeSpecName "kube-api-access-dkpjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:02.845702 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:02.845663 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-openshift-ca\") pod \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\" (UID: \"d1c8f2f5-3602-4e87-b478-bba9e1bcd355\") " Nov 02 20:41:02.845925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:02.845911 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkpjz\" (UniqueName: \"kubernetes.io/projected/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-kube-api-access-dkpjz\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:02.845971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:02.845928 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:02.846026 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:02.846004 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-openshift-ca" pod "d1c8f2f5-3602-4e87-b478-bba9e1bcd355" (UID: "d1c8f2f5-3602-4e87-b478-bba9e1bcd355"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:41:02.946931 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:02.946893 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c8f2f5-3602-4e87-b478-bba9e1bcd355-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:03.545992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:03.545962 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" Nov 02 20:41:03.545992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:03.545969 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc" event={"ID":"d1c8f2f5-3602-4e87-b478-bba9e1bcd355","Type":"ContainerDied","Data":"46bc6ba11212207591dd9b3ad372d9363b4141f26eee4f60a6fe8cfacd29eb02"} Nov 02 20:41:03.546209 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:03.546018 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46bc6ba11212207591dd9b3ad372d9363b4141f26eee4f60a6fe8cfacd29eb02" Nov 02 20:41:07.834006 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:07.833967 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:41:18.237578 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:41:18.237533 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:41:35.105397 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.105357 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42"] Nov 02 20:41:35.105905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.105833 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d1c8f2f5-3602-4e87-b478-bba9e1bcd355" containerName="revoke-keys" Nov 02 20:41:35.105905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.105845 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c8f2f5-3602-4e87-b478-bba9e1bcd355" containerName="revoke-keys" Nov 02 20:41:35.105988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.105937 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c8f2f5-3602-4e87-b478-bba9e1bcd355" containerName="revoke-keys" Nov 02 20:41:35.109475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.109456 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:35.162567 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.162533 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42"] Nov 02 20:41:35.205322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.205270 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:41:35.205322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.205324 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:41:35.205548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.205342 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:41:35.205548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.205350 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:41:35.299431 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.299395 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ab7818-qgv42\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:35.299620 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.299455 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ab7818-qgv42\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:35.299620 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.299595 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpr9p\" (UniqueName: \"kubernetes.io/projected/80b77ce9-1949-4d3c-a578-c0e44d631590-kube-api-access-kpr9p\") pod \"maas-api-revoke-sub-c6ab7818-qgv42\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:35.401173 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.401077 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ab7818-qgv42\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:35.401173 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.401141 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ab7818-qgv42\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:35.401467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.401268 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpr9p\" (UniqueName: \"kubernetes.io/projected/80b77ce9-1949-4d3c-a578-c0e44d631590-kube-api-access-kpr9p\") pod \"maas-api-revoke-sub-c6ab7818-qgv42\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:35.401831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.401809 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6ab7818-qgv42\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:35.403601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.403577 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6ab7818-qgv42\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:35.454458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.454427 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpr9p\" (UniqueName: \"kubernetes.io/projected/80b77ce9-1949-4d3c-a578-c0e44d631590-kube-api-access-kpr9p\") pod \"maas-api-revoke-sub-c6ab7818-qgv42\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:35.722755 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:35.722663 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:36.089338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:36.089307 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42"] Nov 02 20:41:36.090421 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:41:36.090393 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b77ce9_1949_4d3c_a578_c0e44d631590.slice/crio-715176d949cc91b24e9050a74e89a2c7251a004bd4f583d213341655a664269f WatchSource:0}: Error finding container 715176d949cc91b24e9050a74e89a2c7251a004bd4f583d213341655a664269f: Status 404 returned error can't find the container with id 715176d949cc91b24e9050a74e89a2c7251a004bd4f583d213341655a664269f Nov 02 20:41:36.777670 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:36.777578 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" event={"ID":"80b77ce9-1949-4d3c-a578-c0e44d631590","Type":"ContainerStarted","Data":"cba402d0fbc543c53245d74ef16f8e49c0a5f53db22e9475399f117e049d955c"} Nov 02 20:41:36.777670 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:36.777636 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" event={"ID":"80b77ce9-1949-4d3c-a578-c0e44d631590","Type":"ContainerStarted","Data":"715176d949cc91b24e9050a74e89a2c7251a004bd4f583d213341655a664269f"} Nov 02 20:41:36.884345 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:36.884286 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" podStartSLOduration=1.8842590860000001 podStartE2EDuration="1.884259086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:41:36.882129096 +0000 UTC m=+2939.492781186" watchObservedRunningTime="2026-11-02 20:41:36.884259086 +0000 UTC m=+2939.494911178" Nov 02 20:41:37.784989 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:37.784950 5399 generic.go:356] "Generic (PLEG): container finished" podID="80b77ce9-1949-4d3c-a578-c0e44d631590" containerID="cba402d0fbc543c53245d74ef16f8e49c0a5f53db22e9475399f117e049d955c" exitCode=0 Nov 02 20:41:37.785388 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:37.785019 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" event={"ID":"80b77ce9-1949-4d3c-a578-c0e44d631590","Type":"ContainerDied","Data":"cba402d0fbc543c53245d74ef16f8e49c0a5f53db22e9475399f117e049d955c"} Nov 02 20:41:38.944132 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:38.944105 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:39.046390 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.046254 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b77ce9-1949-4d3c-a578-c0e44d631590-kube-api-access-kpr9p\" (UniqueName: \"kubernetes.io/projected/80b77ce9-1949-4d3c-a578-c0e44d631590-kube-api-access-kpr9p\") pod \"80b77ce9-1949-4d3c-a578-c0e44d631590\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " Nov 02 20:41:39.046574 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.046400 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-openshift-ca\") pod \"80b77ce9-1949-4d3c-a578-c0e44d631590\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " Nov 02 20:41:39.046574 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.046531 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-xks-ca\") pod \"80b77ce9-1949-4d3c-a578-c0e44d631590\" (UID: \"80b77ce9-1949-4d3c-a578-c0e44d631590\") " Nov 02 20:41:39.046984 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.046956 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-openshift-ca" pod "80b77ce9-1949-4d3c-a578-c0e44d631590" (UID: "80b77ce9-1949-4d3c-a578-c0e44d631590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:41:39.048626 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.048598 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b77ce9-1949-4d3c-a578-c0e44d631590-kube-api-access-kpr9p" pod "80b77ce9-1949-4d3c-a578-c0e44d631590" (UID: "80b77ce9-1949-4d3c-a578-c0e44d631590"). InnerVolumeSpecName "kube-api-access-kpr9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:39.048773 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.048750 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-xks-ca" pod "80b77ce9-1949-4d3c-a578-c0e44d631590" (UID: "80b77ce9-1949-4d3c-a578-c0e44d631590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:41:39.148100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.148063 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:39.148100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.148099 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpr9p\" (UniqueName: \"kubernetes.io/projected/80b77ce9-1949-4d3c-a578-c0e44d631590-kube-api-access-kpr9p\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:39.148461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.148110 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80b77ce9-1949-4d3c-a578-c0e44d631590-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:39.185883 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.185849 5399 scope.go:122] "RemoveContainer" containerID="fb8e3ccb40c89cc2b909a7c94a150bb8714c625635a915e92780ec704ff41110" Nov 02 20:41:39.195897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.195875 5399 scope.go:122] "RemoveContainer" containerID="064488a7c6fbb55c312777b4929ebd6042772a6c7cfea33c3ccc001d20331ac4" Nov 02 20:41:39.207896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.207873 5399 scope.go:122] "RemoveContainer" containerID="2959119aabc9b8f0fb416222973b1f63727dcfeefa66358d14515e9763685715" Nov 02 20:41:39.217914 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.217893 5399 scope.go:122] "RemoveContainer" containerID="9e99f0dcfab25ed66267544bec2690dc5be7678fa0ad0d981332ecc45a8aede4" Nov 02 20:41:39.800040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.799950 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" Nov 02 20:41:39.800040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.799986 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42" event={"ID":"80b77ce9-1949-4d3c-a578-c0e44d631590","Type":"ContainerDied","Data":"715176d949cc91b24e9050a74e89a2c7251a004bd4f583d213341655a664269f"} Nov 02 20:41:39.800040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:39.800034 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="715176d949cc91b24e9050a74e89a2c7251a004bd4f583d213341655a664269f" Nov 02 20:41:41.601570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.601520 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s"] Nov 02 20:41:41.602073 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.602038 5399 cpu_manager.go:414] "containerMap: removing container" podUID="80b77ce9-1949-4d3c-a578-c0e44d631590" containerName="revoke-keys" Nov 02 20:41:41.602073 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.602052 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b77ce9-1949-4d3c-a578-c0e44d631590" containerName="revoke-keys" Nov 02 20:41:41.602196 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.602161 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b77ce9-1949-4d3c-a578-c0e44d631590" containerName="revoke-keys" Nov 02 20:41:41.607237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.607216 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:41.659623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.658265 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s"] Nov 02 20:41:41.677181 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.677141 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddc3fd7d-8t76s\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:41.677418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.677396 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf88b\" (UniqueName: \"kubernetes.io/projected/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-kube-api-access-pf88b\") pod \"maas-api-revoke-sub-ddc3fd7d-8t76s\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:41.677535 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.677513 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddc3fd7d-8t76s\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:41.700167 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.700134 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:41:41.700408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.700228 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:41:41.700408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.700308 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:41:41.705492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.705471 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:41:41.778992 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.778962 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf88b\" (UniqueName: \"kubernetes.io/projected/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-kube-api-access-pf88b\") pod \"maas-api-revoke-sub-ddc3fd7d-8t76s\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:41.779192 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.779023 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddc3fd7d-8t76s\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:41.779192 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.779055 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddc3fd7d-8t76s\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:41.779763 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.779741 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddc3fd7d-8t76s\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:41.781364 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.781339 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddc3fd7d-8t76s\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:41.832144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.832107 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf88b\" (UniqueName: \"kubernetes.io/projected/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-kube-api-access-pf88b\") pod \"maas-api-revoke-sub-ddc3fd7d-8t76s\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:41.920511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:41.920409 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:42.289400 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:42.289375 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s"] Nov 02 20:41:42.290753 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:41:42.290728 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b64cbd6_0cd3_487a_b1ff_15b0b6392865.slice/crio-a399fbbbb6fcdce7b7f0099939d829ef1b5d5cf4e167b06c4a50c314d61b57fb WatchSource:0}: Error finding container a399fbbbb6fcdce7b7f0099939d829ef1b5d5cf4e167b06c4a50c314d61b57fb: Status 404 returned error can't find the container with id a399fbbbb6fcdce7b7f0099939d829ef1b5d5cf4e167b06c4a50c314d61b57fb Nov 02 20:41:42.820865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:42.820831 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" event={"ID":"9b64cbd6-0cd3-487a-b1ff-15b0b6392865","Type":"ContainerStarted","Data":"99191506fb81b80d1e51f98cfaced5824b5dfe18bf6c6ff07578fc0a1d6cc689"} Nov 02 20:41:42.821323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:42.820890 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" event={"ID":"9b64cbd6-0cd3-487a-b1ff-15b0b6392865","Type":"ContainerStarted","Data":"a399fbbbb6fcdce7b7f0099939d829ef1b5d5cf4e167b06c4a50c314d61b57fb"} Nov 02 20:41:42.928178 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:42.928118 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" podStartSLOduration=1.928103618 podStartE2EDuration="1.928103618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:41:42.926611534 +0000 UTC m=+2945.537263622" watchObservedRunningTime="2026-11-02 20:41:42.928103618 +0000 UTC m=+2945.538755748" Nov 02 20:41:43.633423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:43.633387 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc"] Nov 02 20:41:43.638903 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:43.638875 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7b1839-pt2kc"] Nov 02 20:41:43.828983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:43.828950 5399 generic.go:356] "Generic (PLEG): container finished" podID="9b64cbd6-0cd3-487a-b1ff-15b0b6392865" containerID="99191506fb81b80d1e51f98cfaced5824b5dfe18bf6c6ff07578fc0a1d6cc689" exitCode=0 Nov 02 20:41:43.829491 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:43.829040 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" event={"ID":"9b64cbd6-0cd3-487a-b1ff-15b0b6392865","Type":"ContainerDied","Data":"99191506fb81b80d1e51f98cfaced5824b5dfe18bf6c6ff07578fc0a1d6cc689"} Nov 02 20:41:44.976061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:44.976036 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:45.014399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.014369 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-openshift-ca\") pod \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " Nov 02 20:41:45.014537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.014511 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-xks-ca\") pod \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " Nov 02 20:41:45.014583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.014549 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-kube-api-access-pf88b\" (UniqueName: \"kubernetes.io/projected/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-kube-api-access-pf88b\") pod \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\" (UID: \"9b64cbd6-0cd3-487a-b1ff-15b0b6392865\") " Nov 02 20:41:45.014807 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.014780 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-openshift-ca" pod "9b64cbd6-0cd3-487a-b1ff-15b0b6392865" (UID: "9b64cbd6-0cd3-487a-b1ff-15b0b6392865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:41:45.014991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.014974 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:45.016607 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.016583 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-xks-ca" pod "9b64cbd6-0cd3-487a-b1ff-15b0b6392865" (UID: "9b64cbd6-0cd3-487a-b1ff-15b0b6392865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:41:45.016720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.016697 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-kube-api-access-pf88b" pod "9b64cbd6-0cd3-487a-b1ff-15b0b6392865" (UID: "9b64cbd6-0cd3-487a-b1ff-15b0b6392865"). InnerVolumeSpecName "kube-api-access-pf88b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:45.073390 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.073357 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf"] Nov 02 20:41:45.075694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.075668 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f1dad2-ggcbf"] Nov 02 20:41:45.116444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.116361 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:45.116444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.116392 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf88b\" (UniqueName: \"kubernetes.io/projected/9b64cbd6-0cd3-487a-b1ff-15b0b6392865-kube-api-access-pf88b\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:45.507913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.507841 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c8f2f5-3602-4e87-b478-bba9e1bcd355" path="/var/lib/kubelet/pods/d1c8f2f5-3602-4e87-b478-bba9e1bcd355/volumes" Nov 02 20:41:45.508296 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.508265 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cf1f20-7f73-4775-b070-1f3ce66c8973" path="/var/lib/kubelet/pods/e4cf1f20-7f73-4775-b070-1f3ce66c8973/volumes" Nov 02 20:41:45.843727 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.843698 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" Nov 02 20:41:45.843917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.843737 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s" event={"ID":"9b64cbd6-0cd3-487a-b1ff-15b0b6392865","Type":"ContainerDied","Data":"a399fbbbb6fcdce7b7f0099939d829ef1b5d5cf4e167b06c4a50c314d61b57fb"} Nov 02 20:41:45.843917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:45.843792 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a399fbbbb6fcdce7b7f0099939d829ef1b5d5cf4e167b06c4a50c314d61b57fb" Nov 02 20:41:48.115357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.115315 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s"] Nov 02 20:41:48.116606 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.116572 5399 cpu_manager.go:414] "containerMap: removing container" podUID="9b64cbd6-0cd3-487a-b1ff-15b0b6392865" containerName="revoke-keys" Nov 02 20:41:48.116606 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.116598 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b64cbd6-0cd3-487a-b1ff-15b0b6392865" containerName="revoke-keys" Nov 02 20:41:48.116779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.116685 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b64cbd6-0cd3-487a-b1ff-15b0b6392865" containerName="revoke-keys" Nov 02 20:41:48.121605 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.121583 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.173759 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.173725 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s"] Nov 02 20:41:48.203674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.203632 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:41:48.203873 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.203633 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:41:48.215261 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.215231 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:41:48.215571 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.215549 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:41:48.240603 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:41:48.240560 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:41:48.250446 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.250418 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7774ff05-wqz2s\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.250594 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.250455 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcgg\" (UniqueName: \"kubernetes.io/projected/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-kube-api-access-tvcgg\") pod \"maas-api-revoke-sub-7774ff05-wqz2s\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.250594 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.250492 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7774ff05-wqz2s\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.351925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.351893 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7774ff05-wqz2s\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.352136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.351929 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcgg\" (UniqueName: \"kubernetes.io/projected/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-kube-api-access-tvcgg\") pod \"maas-api-revoke-sub-7774ff05-wqz2s\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.352136 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.351961 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7774ff05-wqz2s\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.352723 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.352695 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7774ff05-wqz2s\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.355678 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.355657 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7774ff05-wqz2s\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.405752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.405674 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcgg\" (UniqueName: \"kubernetes.io/projected/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-kube-api-access-tvcgg\") pod \"maas-api-revoke-sub-7774ff05-wqz2s\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.438820 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.438786 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:48.806423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.806391 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s"] Nov 02 20:41:48.807080 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:41:48.807045 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d65f651_b3e0_4ae1_be4e_8e568705dfdd.slice/crio-24c44fc3219f6c7f3134834829d00122e8955a3fd2c422440b5f1f7cbf1c079a WatchSource:0}: Error finding container 24c44fc3219f6c7f3134834829d00122e8955a3fd2c422440b5f1f7cbf1c079a: Status 404 returned error can't find the container with id 24c44fc3219f6c7f3134834829d00122e8955a3fd2c422440b5f1f7cbf1c079a Nov 02 20:41:48.864448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:48.864418 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" event={"ID":"2d65f651-b3e0-4ae1-be4e-8e568705dfdd","Type":"ContainerStarted","Data":"24c44fc3219f6c7f3134834829d00122e8955a3fd2c422440b5f1f7cbf1c079a"} Nov 02 20:41:49.872244 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:49.872211 5399 generic.go:356] "Generic (PLEG): container finished" podID="2d65f651-b3e0-4ae1-be4e-8e568705dfdd" containerID="88fc518f6d98c6ed09129935fcd13dc4315c6d589562de4fe93d4ce78d61c87a" exitCode=0 Nov 02 20:41:49.872631 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:49.872306 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" event={"ID":"2d65f651-b3e0-4ae1-be4e-8e568705dfdd","Type":"ContainerDied","Data":"88fc518f6d98c6ed09129935fcd13dc4315c6d589562de4fe93d4ce78d61c87a"} Nov 02 20:41:51.022643 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.022612 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:51.028433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.028404 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn"] Nov 02 20:41:51.028888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.028875 5399 cpu_manager.go:414] "containerMap: removing container" podUID="2d65f651-b3e0-4ae1-be4e-8e568705dfdd" containerName="revoke-keys" Nov 02 20:41:51.028933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.028890 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d65f651-b3e0-4ae1-be4e-8e568705dfdd" containerName="revoke-keys" Nov 02 20:41:51.028981 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.028973 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d65f651-b3e0-4ae1-be4e-8e568705dfdd" containerName="revoke-keys" Nov 02 20:41:51.032211 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.032189 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.084689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.084654 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn"] Nov 02 20:41:51.084845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.084751 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-kube-api-access-tvcgg\" (UniqueName: \"kubernetes.io/projected/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-kube-api-access-tvcgg\") pod \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " Nov 02 20:41:51.084845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.084795 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-openshift-ca\") pod \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " Nov 02 20:41:51.085003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.084888 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-xks-ca\") pod \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\" (UID: \"2d65f651-b3e0-4ae1-be4e-8e568705dfdd\") " Nov 02 20:41:51.085237 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.085200 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25f96\" (UniqueName: \"kubernetes.io/projected/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-kube-api-access-25f96\") pod \"maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.085353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.085239 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-openshift-ca" pod "2d65f651-b3e0-4ae1-be4e-8e568705dfdd" (UID: "2d65f651-b3e0-4ae1-be4e-8e568705dfdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:41:51.085353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.085257 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.085438 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.085417 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.085587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.085567 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:51.087019 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.086991 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-kube-api-access-tvcgg" pod "2d65f651-b3e0-4ae1-be4e-8e568705dfdd" (UID: "2d65f651-b3e0-4ae1-be4e-8e568705dfdd"). InnerVolumeSpecName "kube-api-access-tvcgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:51.087414 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.087390 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-xks-ca" pod "2d65f651-b3e0-4ae1-be4e-8e568705dfdd" (UID: "2d65f651-b3e0-4ae1-be4e-8e568705dfdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:41:51.187365 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.187250 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.187523 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.187451 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25f96\" (UniqueName: \"kubernetes.io/projected/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-kube-api-access-25f96\") pod \"maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.187523 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.187498 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.187618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.187560 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvcgg\" (UniqueName: \"kubernetes.io/projected/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-kube-api-access-tvcgg\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:51.187618 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.187576 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d65f651-b3e0-4ae1-be4e-8e568705dfdd-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:51.188312 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.188289 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.189809 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.189779 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.241180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.241143 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25f96\" (UniqueName: \"kubernetes.io/projected/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-kube-api-access-25f96\") pod \"maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.345951 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.345908 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:51.721860 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.721830 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn"] Nov 02 20:41:51.723027 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:41:51.723004 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003aceaa_2c7d_43a0_b9eb_1bc4a47596ea.slice/crio-6be55625fd259d18a78b75fe983b950100174ae7d12b6aa0b787d997339805d5 WatchSource:0}: Error finding container 6be55625fd259d18a78b75fe983b950100174ae7d12b6aa0b787d997339805d5: Status 404 returned error can't find the container with id 6be55625fd259d18a78b75fe983b950100174ae7d12b6aa0b787d997339805d5 Nov 02 20:41:51.884872 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.884838 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" Nov 02 20:41:51.884872 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.884865 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s" event={"ID":"2d65f651-b3e0-4ae1-be4e-8e568705dfdd","Type":"ContainerDied","Data":"24c44fc3219f6c7f3134834829d00122e8955a3fd2c422440b5f1f7cbf1c079a"} Nov 02 20:41:51.885125 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.884916 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24c44fc3219f6c7f3134834829d00122e8955a3fd2c422440b5f1f7cbf1c079a" Nov 02 20:41:51.886305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:51.886252 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" event={"ID":"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea","Type":"ContainerStarted","Data":"6be55625fd259d18a78b75fe983b950100174ae7d12b6aa0b787d997339805d5"} Nov 02 20:41:52.894512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:52.894416 5399 generic.go:356] "Generic (PLEG): container finished" podID="003aceaa-2c7d-43a0-b9eb-1bc4a47596ea" containerID="48227cc9ead8aa035a64ed4a6d08fdc2f689fa47bf273a0d00da8b6c5da58c35" exitCode=0 Nov 02 20:41:52.894512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:52.894498 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" event={"ID":"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea","Type":"ContainerDied","Data":"48227cc9ead8aa035a64ed4a6d08fdc2f689fa47bf273a0d00da8b6c5da58c35"} Nov 02 20:41:54.036694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.036665 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:54.119597 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.119563 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-openshift-ca\") pod \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " Nov 02 20:41:54.119775 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.119603 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-kube-api-access-25f96\" (UniqueName: \"kubernetes.io/projected/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-kube-api-access-25f96\") pod \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " Nov 02 20:41:54.119775 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.119699 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-xks-ca\") pod \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\" (UID: \"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea\") " Nov 02 20:41:54.120003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.119980 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-openshift-ca" pod "003aceaa-2c7d-43a0-b9eb-1bc4a47596ea" (UID: "003aceaa-2c7d-43a0-b9eb-1bc4a47596ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:41:54.120111 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.120095 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:54.121739 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.121719 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-kube-api-access-25f96" pod "003aceaa-2c7d-43a0-b9eb-1bc4a47596ea" (UID: "003aceaa-2c7d-43a0-b9eb-1bc4a47596ea"). InnerVolumeSpecName "kube-api-access-25f96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:54.121838 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.121823 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-xks-ca" pod "003aceaa-2c7d-43a0-b9eb-1bc4a47596ea" (UID: "003aceaa-2c7d-43a0-b9eb-1bc4a47596ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:41:54.221658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.221576 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25f96\" (UniqueName: \"kubernetes.io/projected/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-kube-api-access-25f96\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:54.221658 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.221603 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:54.907512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.907479 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" Nov 02 20:41:54.907512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.907496 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn" event={"ID":"003aceaa-2c7d-43a0-b9eb-1bc4a47596ea","Type":"ContainerDied","Data":"6be55625fd259d18a78b75fe983b950100174ae7d12b6aa0b787d997339805d5"} Nov 02 20:41:54.907711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:54.907542 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6be55625fd259d18a78b75fe983b950100174ae7d12b6aa0b787d997339805d5" Nov 02 20:41:55.545745 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:55.545711 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h"] Nov 02 20:41:59.043563 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:59.043519 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq"] Nov 02 20:41:59.089632 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:59.089588 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b04f4e30-qw8qq"] Nov 02 20:41:59.507337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:59.507304 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0f2396-8f87-4086-98bd-20b5250e157d" path="/var/lib/kubelet/pods/ed0f2396-8f87-4086-98bd-20b5250e157d/volumes" Nov 02 20:41:59.550357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:41:59.550322 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:42:00.984478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:00.984439 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77"] Nov 02 20:42:00.985211 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:00.985191 5399 cpu_manager.go:414] "containerMap: removing container" podUID="003aceaa-2c7d-43a0-b9eb-1bc4a47596ea" containerName="revoke-keys" Nov 02 20:42:00.985257 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:00.985214 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003aceaa-2c7d-43a0-b9eb-1bc4a47596ea" containerName="revoke-keys" Nov 02 20:42:00.985369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:00.985357 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="003aceaa-2c7d-43a0-b9eb-1bc4a47596ea" containerName="revoke-keys" Nov 02 20:42:00.990416 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:00.990394 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.042456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.042423 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77"] Nov 02 20:42:01.085090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.085047 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:42:01.085090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.085076 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:42:01.085327 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.085177 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:42:01.085534 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.085518 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:42:01.102379 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.102350 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-644b8b58-dck77\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.102542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.102523 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-644b8b58-dck77\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.102608 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.102567 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gdmf\" (UniqueName: \"kubernetes.io/projected/356cea95-7878-4474-b71c-75d02dfc9e00-kube-api-access-4gdmf\") pod \"maas-api-revoke-sub-644b8b58-dck77\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.205016 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.204974 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-644b8b58-dck77\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.205184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.205147 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-644b8b58-dck77\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.205236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.205193 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gdmf\" (UniqueName: \"kubernetes.io/projected/356cea95-7878-4474-b71c-75d02dfc9e00-kube-api-access-4gdmf\") pod \"maas-api-revoke-sub-644b8b58-dck77\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.205860 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.205831 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-644b8b58-dck77\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.210945 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.210915 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-644b8b58-dck77\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.258259 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.258185 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gdmf\" (UniqueName: \"kubernetes.io/projected/356cea95-7878-4474-b71c-75d02dfc9e00-kube-api-access-4gdmf\") pod \"maas-api-revoke-sub-644b8b58-dck77\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.309099 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.309062 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:01.674940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.674908 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77"] Nov 02 20:42:01.676945 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:42:01.676908 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356cea95_7878_4474_b71c_75d02dfc9e00.slice/crio-64a9dc14d65e582a57db10796d393b29d58c6dfbaf6d8017273336e831391d29 WatchSource:0}: Error finding container 64a9dc14d65e582a57db10796d393b29d58c6dfbaf6d8017273336e831391d29: Status 404 returned error can't find the container with id 64a9dc14d65e582a57db10796d393b29d58c6dfbaf6d8017273336e831391d29 Nov 02 20:42:01.950628 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.950598 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" event={"ID":"356cea95-7878-4474-b71c-75d02dfc9e00","Type":"ContainerStarted","Data":"9f50efa74372afe2f1466406651e619ebd967a42491d413e3e76b82e4931157b"} Nov 02 20:42:01.950783 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:01.950648 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" event={"ID":"356cea95-7878-4474-b71c-75d02dfc9e00","Type":"ContainerStarted","Data":"64a9dc14d65e582a57db10796d393b29d58c6dfbaf6d8017273336e831391d29"} Nov 02 20:42:02.058316 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:02.058233 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" podStartSLOduration=2.05821829 podStartE2EDuration="2.05821829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:02.055121807 +0000 UTC m=+2964.665773898" watchObservedRunningTime="2026-11-02 20:42:02.05821829 +0000 UTC m=+2964.668870380" Nov 02 20:42:02.958602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:02.958514 5399 generic.go:356] "Generic (PLEG): container finished" podID="356cea95-7878-4474-b71c-75d02dfc9e00" containerID="9f50efa74372afe2f1466406651e619ebd967a42491d413e3e76b82e4931157b" exitCode=0 Nov 02 20:42:02.958760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:02.958598 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" event={"ID":"356cea95-7878-4474-b71c-75d02dfc9e00","Type":"ContainerDied","Data":"9f50efa74372afe2f1466406651e619ebd967a42491d413e3e76b82e4931157b"} Nov 02 20:42:04.105687 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.105660 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:04.242132 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.242044 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356cea95-7878-4474-b71c-75d02dfc9e00-kube-api-access-4gdmf\" (UniqueName: \"kubernetes.io/projected/356cea95-7878-4474-b71c-75d02dfc9e00-kube-api-access-4gdmf\") pod \"356cea95-7878-4474-b71c-75d02dfc9e00\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " Nov 02 20:42:04.242309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.242217 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-openshift-ca\") pod \"356cea95-7878-4474-b71c-75d02dfc9e00\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " Nov 02 20:42:04.242389 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.242370 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-xks-ca\") pod \"356cea95-7878-4474-b71c-75d02dfc9e00\" (UID: \"356cea95-7878-4474-b71c-75d02dfc9e00\") " Nov 02 20:42:04.242645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.242612 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-openshift-ca" pod "356cea95-7878-4474-b71c-75d02dfc9e00" (UID: "356cea95-7878-4474-b71c-75d02dfc9e00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:04.242789 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.242780 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:04.244397 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.244371 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356cea95-7878-4474-b71c-75d02dfc9e00-kube-api-access-4gdmf" pod "356cea95-7878-4474-b71c-75d02dfc9e00" (UID: "356cea95-7878-4474-b71c-75d02dfc9e00"). InnerVolumeSpecName "kube-api-access-4gdmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:04.244855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.244827 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-xks-ca" pod "356cea95-7878-4474-b71c-75d02dfc9e00" (UID: "356cea95-7878-4474-b71c-75d02dfc9e00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:04.344039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.344001 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gdmf\" (UniqueName: \"kubernetes.io/projected/356cea95-7878-4474-b71c-75d02dfc9e00-kube-api-access-4gdmf\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:04.344039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.344036 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356cea95-7878-4474-b71c-75d02dfc9e00-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:04.972204 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.972175 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" Nov 02 20:42:04.972428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.972220 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77" event={"ID":"356cea95-7878-4474-b71c-75d02dfc9e00","Type":"ContainerDied","Data":"64a9dc14d65e582a57db10796d393b29d58c6dfbaf6d8017273336e831391d29"} Nov 02 20:42:04.972428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:04.972304 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64a9dc14d65e582a57db10796d393b29d58c6dfbaf6d8017273336e831391d29" Nov 02 20:42:06.139942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.139905 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf"] Nov 02 20:42:06.140463 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.140420 5399 cpu_manager.go:414] "containerMap: removing container" podUID="356cea95-7878-4474-b71c-75d02dfc9e00" containerName="revoke-keys" Nov 02 20:42:06.140463 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.140432 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356cea95-7878-4474-b71c-75d02dfc9e00" containerName="revoke-keys" Nov 02 20:42:06.140548 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.140497 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="356cea95-7878-4474-b71c-75d02dfc9e00" containerName="revoke-keys" Nov 02 20:42:06.143813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.143792 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.161806 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.161781 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfc2\" (UniqueName: \"kubernetes.io/projected/246aea8c-620e-49ca-9507-9e5ce27202f5-kube-api-access-rqfc2\") pod \"maas-api-revoke-sub-cab50db4-jrsgf\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.162034 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.161852 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab50db4-jrsgf\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.162034 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.161922 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab50db4-jrsgf\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.193871 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.193840 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2"] Nov 02 20:42:06.198498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.198468 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.236866 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.236829 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:42:06.237037 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.236893 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:42:06.239550 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.239522 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:42:06.239831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.239815 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:42:06.247734 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.244336 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf"] Nov 02 20:42:06.256786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.256749 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2"] Nov 02 20:42:06.262879 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.262854 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53183fcb-xlch2\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.263010 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.262890 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfc2\" (UniqueName: \"kubernetes.io/projected/246aea8c-620e-49ca-9507-9e5ce27202f5-kube-api-access-rqfc2\") pod \"maas-api-revoke-sub-cab50db4-jrsgf\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.263137 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.263109 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab50db4-jrsgf\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.263487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.263164 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab50db4-jrsgf\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.263487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.263232 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpc4\" (UniqueName: \"kubernetes.io/projected/d29d801d-a2f2-4886-b9a6-06abba1407f7-kube-api-access-5xpc4\") pod \"maas-api-revoke-sub-53183fcb-xlch2\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.263487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.263332 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53183fcb-xlch2\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.263845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.263820 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cab50db4-jrsgf\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.265726 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.265698 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cab50db4-jrsgf\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.315997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.315958 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfc2\" (UniqueName: \"kubernetes.io/projected/246aea8c-620e-49ca-9507-9e5ce27202f5-kube-api-access-rqfc2\") pod \"maas-api-revoke-sub-cab50db4-jrsgf\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.364882 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.364841 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53183fcb-xlch2\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.365074 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.364967 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpc4\" (UniqueName: \"kubernetes.io/projected/d29d801d-a2f2-4886-b9a6-06abba1407f7-kube-api-access-5xpc4\") pod \"maas-api-revoke-sub-53183fcb-xlch2\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.365074 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.365011 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53183fcb-xlch2\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.365556 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.365531 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53183fcb-xlch2\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.367357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.367335 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53183fcb-xlch2\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.417798 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.417712 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpc4\" (UniqueName: \"kubernetes.io/projected/d29d801d-a2f2-4886-b9a6-06abba1407f7-kube-api-access-5xpc4\") pod \"maas-api-revoke-sub-53183fcb-xlch2\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.457825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.457790 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:06.515533 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.515493 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:06.826934 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.826904 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf"] Nov 02 20:42:06.827849 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:42:06.827823 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246aea8c_620e_49ca_9507_9e5ce27202f5.slice/crio-f7d10388c1cf7df970b95e7ef1519badbd709557d2845f2a5fe0d089f855ef0f WatchSource:0}: Error finding container f7d10388c1cf7df970b95e7ef1519badbd709557d2845f2a5fe0d089f855ef0f: Status 404 returned error can't find the container with id f7d10388c1cf7df970b95e7ef1519badbd709557d2845f2a5fe0d089f855ef0f Nov 02 20:42:06.921593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.921567 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2"] Nov 02 20:42:06.922452 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:42:06.922427 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29d801d_a2f2_4886_b9a6_06abba1407f7.slice/crio-f21f8f2fe05ab88efc0f04be38293e2306d9b8d76bc4f4092f62f23dd9027e0a WatchSource:0}: Error finding container f21f8f2fe05ab88efc0f04be38293e2306d9b8d76bc4f4092f62f23dd9027e0a: Status 404 returned error can't find the container with id f21f8f2fe05ab88efc0f04be38293e2306d9b8d76bc4f4092f62f23dd9027e0a Nov 02 20:42:06.989008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.988967 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" event={"ID":"d29d801d-a2f2-4886-b9a6-06abba1407f7","Type":"ContainerStarted","Data":"f21f8f2fe05ab88efc0f04be38293e2306d9b8d76bc4f4092f62f23dd9027e0a"} Nov 02 20:42:06.990427 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:06.990394 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" event={"ID":"246aea8c-620e-49ca-9507-9e5ce27202f5","Type":"ContainerStarted","Data":"f7d10388c1cf7df970b95e7ef1519badbd709557d2845f2a5fe0d089f855ef0f"} Nov 02 20:42:07.998434 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:07.998343 5399 generic.go:356] "Generic (PLEG): container finished" podID="d29d801d-a2f2-4886-b9a6-06abba1407f7" containerID="a5c39c4f2c8cbae4d48678dfe5fd799a9ed3cd2ad7f2e628d92c7a014fa075df" exitCode=0 Nov 02 20:42:07.998898 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:07.998429 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" event={"ID":"d29d801d-a2f2-4886-b9a6-06abba1407f7","Type":"ContainerDied","Data":"a5c39c4f2c8cbae4d48678dfe5fd799a9ed3cd2ad7f2e628d92c7a014fa075df"} Nov 02 20:42:08.000046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:08.000022 5399 generic.go:356] "Generic (PLEG): container finished" podID="246aea8c-620e-49ca-9507-9e5ce27202f5" containerID="f3f8d92f910494b64544836beedbc96a22af881f3185df91a260e09bb834ca2b" exitCode=0 Nov 02 20:42:08.000156 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:08.000102 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" event={"ID":"246aea8c-620e-49ca-9507-9e5ce27202f5","Type":"ContainerDied","Data":"f3f8d92f910494b64544836beedbc96a22af881f3185df91a260e09bb834ca2b"} Nov 02 20:42:09.166432 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.166396 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:09.190317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.190257 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:09.194637 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.194615 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-xks-ca\") pod \"d29d801d-a2f2-4886-b9a6-06abba1407f7\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " Nov 02 20:42:09.194744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.194682 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-openshift-ca\") pod \"d29d801d-a2f2-4886-b9a6-06abba1407f7\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " Nov 02 20:42:09.194744 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.194713 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d29d801d-a2f2-4886-b9a6-06abba1407f7-kube-api-access-5xpc4\" (UniqueName: \"kubernetes.io/projected/d29d801d-a2f2-4886-b9a6-06abba1407f7-kube-api-access-5xpc4\") pod \"d29d801d-a2f2-4886-b9a6-06abba1407f7\" (UID: \"d29d801d-a2f2-4886-b9a6-06abba1407f7\") " Nov 02 20:42:09.195075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.195048 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-openshift-ca" pod "d29d801d-a2f2-4886-b9a6-06abba1407f7" (UID: "d29d801d-a2f2-4886-b9a6-06abba1407f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:09.196815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.196785 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-xks-ca" pod "d29d801d-a2f2-4886-b9a6-06abba1407f7" (UID: "d29d801d-a2f2-4886-b9a6-06abba1407f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:09.196919 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.196841 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29d801d-a2f2-4886-b9a6-06abba1407f7-kube-api-access-5xpc4" pod "d29d801d-a2f2-4886-b9a6-06abba1407f7" (UID: "d29d801d-a2f2-4886-b9a6-06abba1407f7"). InnerVolumeSpecName "kube-api-access-5xpc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:09.295922 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.295829 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-openshift-ca\") pod \"246aea8c-620e-49ca-9507-9e5ce27202f5\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " Nov 02 20:42:09.295922 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.295894 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246aea8c-620e-49ca-9507-9e5ce27202f5-kube-api-access-rqfc2\" (UniqueName: \"kubernetes.io/projected/246aea8c-620e-49ca-9507-9e5ce27202f5-kube-api-access-rqfc2\") pod \"246aea8c-620e-49ca-9507-9e5ce27202f5\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " Nov 02 20:42:09.296161 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.295979 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-xks-ca\") pod \"246aea8c-620e-49ca-9507-9e5ce27202f5\" (UID: \"246aea8c-620e-49ca-9507-9e5ce27202f5\") " Nov 02 20:42:09.296303 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.296243 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-openshift-ca" pod "246aea8c-620e-49ca-9507-9e5ce27202f5" (UID: "246aea8c-620e-49ca-9507-9e5ce27202f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:09.296709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.296639 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:09.296709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.296666 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:09.296709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.296685 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29d801d-a2f2-4886-b9a6-06abba1407f7-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:09.296709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.296699 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xpc4\" (UniqueName: \"kubernetes.io/projected/d29d801d-a2f2-4886-b9a6-06abba1407f7-kube-api-access-5xpc4\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:09.298185 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.298164 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-xks-ca" pod "246aea8c-620e-49ca-9507-9e5ce27202f5" (UID: "246aea8c-620e-49ca-9507-9e5ce27202f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:09.298606 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.298590 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246aea8c-620e-49ca-9507-9e5ce27202f5-kube-api-access-rqfc2" pod "246aea8c-620e-49ca-9507-9e5ce27202f5" (UID: "246aea8c-620e-49ca-9507-9e5ce27202f5"). InnerVolumeSpecName "kube-api-access-rqfc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:09.397849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.397817 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246aea8c-620e-49ca-9507-9e5ce27202f5-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:09.397849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:09.397847 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqfc2\" (UniqueName: \"kubernetes.io/projected/246aea8c-620e-49ca-9507-9e5ce27202f5-kube-api-access-rqfc2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:10.014200 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.014171 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" Nov 02 20:42:10.014442 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.014346 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2" event={"ID":"d29d801d-a2f2-4886-b9a6-06abba1407f7","Type":"ContainerDied","Data":"f21f8f2fe05ab88efc0f04be38293e2306d9b8d76bc4f4092f62f23dd9027e0a"} Nov 02 20:42:10.014442 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.014408 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f21f8f2fe05ab88efc0f04be38293e2306d9b8d76bc4f4092f62f23dd9027e0a" Nov 02 20:42:10.015785 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.015764 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" Nov 02 20:42:10.015910 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.015764 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf" event={"ID":"246aea8c-620e-49ca-9507-9e5ce27202f5","Type":"ContainerDied","Data":"f7d10388c1cf7df970b95e7ef1519badbd709557d2845f2a5fe0d089f855ef0f"} Nov 02 20:42:10.015910 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.015889 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7d10388c1cf7df970b95e7ef1519badbd709557d2845f2a5fe0d089f855ef0f" Nov 02 20:42:10.566153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.566105 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" containerID="cri-o://d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb" gracePeriod=45 Nov 02 20:42:10.823464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.823400 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:42:10.914619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.914583 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tls-certs\") pod \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " Nov 02 20:42:10.914808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.914686 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-dshm\") pod \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " Nov 02 20:42:10.914886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.914860 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-model-cache\") pod \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " Nov 02 20:42:10.914980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.914965 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-home\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-home\") pod \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " Nov 02 20:42:10.915054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.915026 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-kube-api-access-rb94p\" (UniqueName: \"kubernetes.io/projected/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-kube-api-access-rb94p\") pod \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " Nov 02 20:42:10.915153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.915139 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tmp-dir\") pod \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\" (UID: \"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6\") " Nov 02 20:42:10.915214 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.915142 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-model-cache" pod "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" (UID: "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:10.915393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.915370 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-home" pod "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" (UID: "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:10.915572 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.915550 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tmp-dir" pod "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" (UID: "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:10.915645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.915576 5399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-model-cache\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:10.915645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.915596 5399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-home\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:10.917111 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.917081 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-dshm" pod "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" (UID: "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:10.917111 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.917089 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tls-certs" pod "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" (UID: "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:10.917249 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:10.917136 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-kube-api-access-rb94p" pod "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" (UID: "e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6"). InnerVolumeSpecName "kube-api-access-rb94p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:11.016630 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.016591 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb94p\" (UniqueName: \"kubernetes.io/projected/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-kube-api-access-rb94p\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.016630 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.016629 5399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tmp-dir\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.016842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.016646 5399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-tls-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.016842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.016659 5399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6-dshm\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:11.023652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.023621 5399 generic.go:356] "Generic (PLEG): container finished" podID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerID="d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb" exitCode=0 Nov 02 20:42:11.023826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.023677 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" Nov 02 20:42:11.023826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.023709 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" event={"ID":"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6","Type":"ContainerDied","Data":"d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb"} Nov 02 20:42:11.023826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.023762 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h" event={"ID":"e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6","Type":"ContainerDied","Data":"c1596d4afd2ba4de83df0700f7c109b3852e2bd0d4dc56bc9690965f9a8073f5"} Nov 02 20:42:11.023826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.023786 5399 scope.go:122] "RemoveContainer" containerID="d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb" Nov 02 20:42:11.038453 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.038402 5399 scope.go:122] "RemoveContainer" containerID="d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb" Nov 02 20:42:11.038751 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:42:11.038730 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb\": container with ID starting with d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb not found: ID does not exist" containerID="d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb" Nov 02 20:42:11.038839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.038766 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb"} err="failed to get container status \"d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb\": rpc error: code = NotFound desc = could not find container \"d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb\": container with ID starting with d4b53b40f9f232d0e58bda2cec63ad321ddddc047fbb0fff624790af497a13eb not found: ID does not exist" Nov 02 20:42:11.177183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.177095 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h"] Nov 02 20:42:11.179667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.179642 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ae8e6e/premium-w5-ae8e6e-kserve-7cc5889f7c-77w2h"] Nov 02 20:42:11.507324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:11.507221 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" path="/var/lib/kubelet/pods/e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6/volumes" Nov 02 20:42:18.237540 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:42:18.237448 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:42:27.520953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.520920 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw"] Nov 02 20:42:27.521404 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.521389 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" Nov 02 20:42:27.521453 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.521406 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" Nov 02 20:42:27.521453 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.521441 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d29d801d-a2f2-4886-b9a6-06abba1407f7" containerName="revoke-keys" Nov 02 20:42:27.521453 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.521446 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29d801d-a2f2-4886-b9a6-06abba1407f7" containerName="revoke-keys" Nov 02 20:42:27.521547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.521458 5399 cpu_manager.go:414] "containerMap: removing container" podUID="246aea8c-620e-49ca-9507-9e5ce27202f5" containerName="revoke-keys" Nov 02 20:42:27.521547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.521463 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246aea8c-620e-49ca-9507-9e5ce27202f5" containerName="revoke-keys" Nov 02 20:42:27.521547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.521540 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e7ae1b-8a2c-4fa1-90d2-42c6df2309a6" containerName="main" Nov 02 20:42:27.521673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.521552 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29d801d-a2f2-4886-b9a6-06abba1407f7" containerName="revoke-keys" Nov 02 20:42:27.521673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.521560 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="246aea8c-620e-49ca-9507-9e5ce27202f5" containerName="revoke-keys" Nov 02 20:42:27.526343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.526323 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:27.564349 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.564319 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw"] Nov 02 20:42:27.576103 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.576059 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:42:27.576260 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.576230 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:42:27.608692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.608661 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:42:27.608692 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.608662 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:42:27.719096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.719038 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:27.719323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.719230 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm9z9\" (UniqueName: \"kubernetes.io/projected/606fba13-eb4f-4282-ab23-909c022fc2eb-kube-api-access-cm9z9\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:27.719377 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.719327 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:27.820909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.820861 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm9z9\" (UniqueName: \"kubernetes.io/projected/606fba13-eb4f-4282-ab23-909c022fc2eb-kube-api-access-cm9z9\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:27.821090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.820927 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:27.821090 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.820987 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:27.821689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.821666 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:27.823328 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.823304 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:27.873984 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:27.873952 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm9z9\" (UniqueName: \"kubernetes.io/projected/606fba13-eb4f-4282-ab23-909c022fc2eb-kube-api-access-cm9z9\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:28.143884 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:28.143790 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:28.530199 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:28.530167 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw"] Nov 02 20:42:28.531211 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:42:28.531182 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606fba13_eb4f_4282_ab23_909c022fc2eb.slice/crio-2a1fa60262cad6ae998183d8acd40ddda7edc6b81f81cb6dea7f62efa87980d8 WatchSource:0}: Error finding container 2a1fa60262cad6ae998183d8acd40ddda7edc6b81f81cb6dea7f62efa87980d8: Status 404 returned error can't find the container with id 2a1fa60262cad6ae998183d8acd40ddda7edc6b81f81cb6dea7f62efa87980d8 Nov 02 20:42:29.149701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:29.149596 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" event={"ID":"606fba13-eb4f-4282-ab23-909c022fc2eb","Type":"ContainerStarted","Data":"527a3deb500556eb77c43aa25b7913c262ff093f311feeab96fa56d97ae77ea4"} Nov 02 20:42:29.149701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:29.149667 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" event={"ID":"606fba13-eb4f-4282-ab23-909c022fc2eb","Type":"ContainerStarted","Data":"2a1fa60262cad6ae998183d8acd40ddda7edc6b81f81cb6dea7f62efa87980d8"} Nov 02 20:42:29.255350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:29.255256 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" podStartSLOduration=2.255240984 podStartE2EDuration="2.255240984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:29.253808487 +0000 UTC m=+2991.864460581" watchObservedRunningTime="2026-11-02 20:42:29.255240984 +0000 UTC m=+2991.865893106" Nov 02 20:42:30.158263 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:30.158229 5399 generic.go:356] "Generic (PLEG): container finished" podID="606fba13-eb4f-4282-ab23-909c022fc2eb" containerID="527a3deb500556eb77c43aa25b7913c262ff093f311feeab96fa56d97ae77ea4" exitCode=0 Nov 02 20:42:30.158769 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:30.158269 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" event={"ID":"606fba13-eb4f-4282-ab23-909c022fc2eb","Type":"ContainerDied","Data":"527a3deb500556eb77c43aa25b7913c262ff093f311feeab96fa56d97ae77ea4"} Nov 02 20:42:31.309977 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:31.309953 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:31.356235 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:31.356200 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-openshift-ca\") pod \"606fba13-eb4f-4282-ab23-909c022fc2eb\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " Nov 02 20:42:31.356424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:31.356310 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606fba13-eb4f-4282-ab23-909c022fc2eb-kube-api-access-cm9z9\" (UniqueName: \"kubernetes.io/projected/606fba13-eb4f-4282-ab23-909c022fc2eb-kube-api-access-cm9z9\") pod \"606fba13-eb4f-4282-ab23-909c022fc2eb\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " Nov 02 20:42:31.356424 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:31.356352 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-xks-ca\") pod \"606fba13-eb4f-4282-ab23-909c022fc2eb\" (UID: \"606fba13-eb4f-4282-ab23-909c022fc2eb\") " Nov 02 20:42:31.356662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:31.356635 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-openshift-ca" pod "606fba13-eb4f-4282-ab23-909c022fc2eb" (UID: "606fba13-eb4f-4282-ab23-909c022fc2eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:31.358579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:31.358549 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606fba13-eb4f-4282-ab23-909c022fc2eb-kube-api-access-cm9z9" pod "606fba13-eb4f-4282-ab23-909c022fc2eb" (UID: "606fba13-eb4f-4282-ab23-909c022fc2eb"). InnerVolumeSpecName "kube-api-access-cm9z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:31.358579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:31.358548 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-xks-ca" pod "606fba13-eb4f-4282-ab23-909c022fc2eb" (UID: "606fba13-eb4f-4282-ab23-909c022fc2eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:31.457705 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:31.457628 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm9z9\" (UniqueName: \"kubernetes.io/projected/606fba13-eb4f-4282-ab23-909c022fc2eb-kube-api-access-cm9z9\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:31.457705 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:31.457658 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:31.457705 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:31.457668 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606fba13-eb4f-4282-ab23-909c022fc2eb-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:32.174108 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:32.174061 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" event={"ID":"606fba13-eb4f-4282-ab23-909c022fc2eb","Type":"ContainerDied","Data":"2a1fa60262cad6ae998183d8acd40ddda7edc6b81f81cb6dea7f62efa87980d8"} Nov 02 20:42:32.174413 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:32.174079 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw" Nov 02 20:42:32.174413 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:32.174167 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a1fa60262cad6ae998183d8acd40ddda7edc6b81f81cb6dea7f62efa87980d8" Nov 02 20:42:39.123603 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.123565 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm"] Nov 02 20:42:39.124363 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.124345 5399 cpu_manager.go:414] "containerMap: removing container" podUID="606fba13-eb4f-4282-ab23-909c022fc2eb" containerName="revoke-keys" Nov 02 20:42:39.124425 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.124366 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606fba13-eb4f-4282-ab23-909c022fc2eb" containerName="revoke-keys" Nov 02 20:42:39.124493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.124482 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="606fba13-eb4f-4282-ab23-909c022fc2eb" containerName="revoke-keys" Nov 02 20:42:39.128336 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.128312 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.179312 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.178286 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:42:39.179312 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.178360 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:42:39.179312 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.178423 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:42:39.179312 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.178435 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:42:39.228210 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.228178 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm"] Nov 02 20:42:39.240893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.240863 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72161e39-06e8-4506-840a-701972593d9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.241081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.241033 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72161e39-06e8-4506-840a-701972593d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.241144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.241098 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9tv\" (UniqueName: \"kubernetes.io/projected/72161e39-06e8-4506-840a-701972593d9a-kube-api-access-mv9tv\") pod \"maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.259634 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.259601 5399 scope.go:122] "RemoveContainer" containerID="9ba49c54e4a12bb57aa1b138e1cc8e6e8f6b8aff32323716b4a930f35a9ed817" Nov 02 20:42:39.269912 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.269890 5399 scope.go:122] "RemoveContainer" containerID="3bf0132afae16a225a363d95d290be7da975d247623266f864fada6a38e3f7ee" Nov 02 20:42:39.342585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.342543 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72161e39-06e8-4506-840a-701972593d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.342766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.342597 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9tv\" (UniqueName: \"kubernetes.io/projected/72161e39-06e8-4506-840a-701972593d9a-kube-api-access-mv9tv\") pod \"maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.342766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.342662 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72161e39-06e8-4506-840a-701972593d9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.343238 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.343213 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72161e39-06e8-4506-840a-701972593d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.345003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.344978 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72161e39-06e8-4506-840a-701972593d9a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.395664 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.395596 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9tv\" (UniqueName: \"kubernetes.io/projected/72161e39-06e8-4506-840a-701972593d9a-kube-api-access-mv9tv\") pod \"maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.442419 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.442385 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:39.609739 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.609561 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4"] Nov 02 20:42:39.833642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:39.833599 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm"] Nov 02 20:42:39.835663 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:42:39.835632 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72161e39_06e8_4506_840a_701972593d9a.slice/crio-80d5164b3808d59d42c44ddb85390a5c7ec46d0736cc62955f455845a972692d WatchSource:0}: Error finding container 80d5164b3808d59d42c44ddb85390a5c7ec46d0736cc62955f455845a972692d: Status 404 returned error can't find the container with id 80d5164b3808d59d42c44ddb85390a5c7ec46d0736cc62955f455845a972692d Nov 02 20:42:40.227772 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:40.227738 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" event={"ID":"72161e39-06e8-4506-840a-701972593d9a","Type":"ContainerStarted","Data":"a0ac1e81dd04fe4934ed023a3df95e54012ec013d7707d1815589e891ff0d79d"} Nov 02 20:42:40.228208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:40.227796 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" event={"ID":"72161e39-06e8-4506-840a-701972593d9a","Type":"ContainerStarted","Data":"80d5164b3808d59d42c44ddb85390a5c7ec46d0736cc62955f455845a972692d"} Nov 02 20:42:40.334458 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:40.334399 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" podStartSLOduration=1.334383192 podStartE2EDuration="1.334383192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:40.332415749 +0000 UTC m=+3002.943067840" watchObservedRunningTime="2026-11-02 20:42:40.334383192 +0000 UTC m=+3002.945035296" Nov 02 20:42:41.236600 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:41.236565 5399 generic.go:356] "Generic (PLEG): container finished" podID="72161e39-06e8-4506-840a-701972593d9a" containerID="a0ac1e81dd04fe4934ed023a3df95e54012ec013d7707d1815589e891ff0d79d" exitCode=0 Nov 02 20:42:41.237083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:41.236642 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" event={"ID":"72161e39-06e8-4506-840a-701972593d9a","Type":"ContainerDied","Data":"a0ac1e81dd04fe4934ed023a3df95e54012ec013d7707d1815589e891ff0d79d"} Nov 02 20:42:42.394909 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.394875 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:42.479688 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.479652 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72161e39-06e8-4506-840a-701972593d9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72161e39-06e8-4506-840a-701972593d9a-maas-api-xks-ca\") pod \"72161e39-06e8-4506-840a-701972593d9a\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " Nov 02 20:42:42.479890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.479691 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72161e39-06e8-4506-840a-701972593d9a-kube-api-access-mv9tv\" (UniqueName: \"kubernetes.io/projected/72161e39-06e8-4506-840a-701972593d9a-kube-api-access-mv9tv\") pod \"72161e39-06e8-4506-840a-701972593d9a\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " Nov 02 20:42:42.479890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.479838 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72161e39-06e8-4506-840a-701972593d9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72161e39-06e8-4506-840a-701972593d9a-maas-api-openshift-ca\") pod \"72161e39-06e8-4506-840a-701972593d9a\" (UID: \"72161e39-06e8-4506-840a-701972593d9a\") " Nov 02 20:42:42.480201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.480176 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72161e39-06e8-4506-840a-701972593d9a-maas-api-openshift-ca" pod "72161e39-06e8-4506-840a-701972593d9a" (UID: "72161e39-06e8-4506-840a-701972593d9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:42.481746 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.481724 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72161e39-06e8-4506-840a-701972593d9a-maas-api-xks-ca" pod "72161e39-06e8-4506-840a-701972593d9a" (UID: "72161e39-06e8-4506-840a-701972593d9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:42.481888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.481868 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72161e39-06e8-4506-840a-701972593d9a-kube-api-access-mv9tv" pod "72161e39-06e8-4506-840a-701972593d9a" (UID: "72161e39-06e8-4506-840a-701972593d9a"). InnerVolumeSpecName "kube-api-access-mv9tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:42.576311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.576265 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn"] Nov 02 20:42:42.579905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.579880 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ae8e6e-7kdcn"] Nov 02 20:42:42.581572 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.581549 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72161e39-06e8-4506-840a-701972593d9a-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:42.581665 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.581575 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72161e39-06e8-4506-840a-701972593d9a-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:42.581665 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:42.581593 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv9tv\" (UniqueName: \"kubernetes.io/projected/72161e39-06e8-4506-840a-701972593d9a-kube-api-access-mv9tv\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:43.250553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:43.250520 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" Nov 02 20:42:43.250553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:43.250542 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm" event={"ID":"72161e39-06e8-4506-840a-701972593d9a","Type":"ContainerDied","Data":"80d5164b3808d59d42c44ddb85390a5c7ec46d0736cc62955f455845a972692d"} Nov 02 20:42:43.250787 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:43.250594 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d5164b3808d59d42c44ddb85390a5c7ec46d0736cc62955f455845a972692d" Nov 02 20:42:43.511068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:43.510991 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003aceaa-2c7d-43a0-b9eb-1bc4a47596ea" path="/var/lib/kubelet/pods/003aceaa-2c7d-43a0-b9eb-1bc4a47596ea/volumes" Nov 02 20:42:44.610941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:44.610877 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" podUID="4b0f654d-dc58-4977-a3a0-b7d35b280f7c" containerName="payload-pre-processing" containerID="cri-o://c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94" gracePeriod=25 Nov 02 20:42:45.508701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:45.508671 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw"] Nov 02 20:42:45.508857 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:45.508710 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9ec0-jhdzw"] Nov 02 20:42:46.429721 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:46.429685 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:42:47.508245 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:47.508214 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606fba13-eb4f-4282-ab23-909c022fc2eb" path="/var/lib/kubelet/pods/606fba13-eb4f-4282-ab23-909c022fc2eb/volumes" Nov 02 20:42:48.237347 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:42:48.237310 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:42:50.863583 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:50.863542 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" podUID="4b0f654d-dc58-4977-a3a0-b7d35b280f7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:42:53.142966 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.142929 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6"] Nov 02 20:42:53.143430 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.143415 5399 cpu_manager.go:414] "containerMap: removing container" podUID="72161e39-06e8-4506-840a-701972593d9a" containerName="revoke-keys" Nov 02 20:42:53.143479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.143430 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72161e39-06e8-4506-840a-701972593d9a" containerName="revoke-keys" Nov 02 20:42:53.143533 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.143524 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="72161e39-06e8-4506-840a-701972593d9a" containerName="revoke-keys" Nov 02 20:42:53.147999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.147973 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:53.200969 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.200932 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6"] Nov 02 20:42:53.242780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.242746 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:42:53.242780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.242760 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:42:53.243686 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.243670 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:42:53.243738 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.243690 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:42:53.309196 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.309156 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:53.309395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.309207 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4gbx\" (UniqueName: \"kubernetes.io/projected/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-kube-api-access-p4gbx\") pod \"maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:53.309395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.309384 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:53.410827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.410740 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:53.410827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.410791 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4gbx\" (UniqueName: \"kubernetes.io/projected/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-kube-api-access-p4gbx\") pod \"maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:53.411030 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.410869 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:53.411511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.411488 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:53.413170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.413145 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:53.463408 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.463379 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4gbx\" (UniqueName: \"kubernetes.io/projected/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-kube-api-access-p4gbx\") pod \"maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:53.762651 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:53.762557 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:54.129659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:54.129627 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6"] Nov 02 20:42:54.131256 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:42:54.131229 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a62275_1d4e_4a65_84e0_ced4b6d3b39b.slice/crio-40104fe4ee1badfe0277d3bada6438d6957a541987e9b285871c1076cb2773ec WatchSource:0}: Error finding container 40104fe4ee1badfe0277d3bada6438d6957a541987e9b285871c1076cb2773ec: Status 404 returned error can't find the container with id 40104fe4ee1badfe0277d3bada6438d6957a541987e9b285871c1076cb2773ec Nov 02 20:42:54.325324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:54.325284 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" event={"ID":"18a62275-1d4e-4a65-84e0-ced4b6d3b39b","Type":"ContainerStarted","Data":"40104fe4ee1badfe0277d3bada6438d6957a541987e9b285871c1076cb2773ec"} Nov 02 20:42:55.333004 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:55.332966 5399 generic.go:356] "Generic (PLEG): container finished" podID="18a62275-1d4e-4a65-84e0-ced4b6d3b39b" containerID="cdf1a73c58aa32667ff1d85e089804440a2637854ad4888aa6c53b756d5c3a00" exitCode=0 Nov 02 20:42:55.333476 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:55.333051 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" event={"ID":"18a62275-1d4e-4a65-84e0-ced4b6d3b39b","Type":"ContainerDied","Data":"cdf1a73c58aa32667ff1d85e089804440a2637854ad4888aa6c53b756d5c3a00"} Nov 02 20:42:55.663632 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:55.663537 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm"] Nov 02 20:42:55.666960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:55.666927 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb78ba-848tm"] Nov 02 20:42:56.479625 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:56.479600 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:56.648578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:56.648490 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-xks-ca\") pod \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " Nov 02 20:42:56.648578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:56.648547 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-kube-api-access-p4gbx\" (UniqueName: \"kubernetes.io/projected/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-kube-api-access-p4gbx\") pod \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " Nov 02 20:42:56.648789 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:56.648585 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-openshift-ca\") pod \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\" (UID: \"18a62275-1d4e-4a65-84e0-ced4b6d3b39b\") " Nov 02 20:42:56.649007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:56.648973 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-openshift-ca" pod "18a62275-1d4e-4a65-84e0-ced4b6d3b39b" (UID: "18a62275-1d4e-4a65-84e0-ced4b6d3b39b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:56.650582 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:56.650557 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-xks-ca" pod "18a62275-1d4e-4a65-84e0-ced4b6d3b39b" (UID: "18a62275-1d4e-4a65-84e0-ced4b6d3b39b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:42:56.650701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:56.650680 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-kube-api-access-p4gbx" pod "18a62275-1d4e-4a65-84e0-ced4b6d3b39b" (UID: "18a62275-1d4e-4a65-84e0-ced4b6d3b39b"). InnerVolumeSpecName "kube-api-access-p4gbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:56.749968 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:56.749931 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:56.749968 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:56.749963 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4gbx\" (UniqueName: \"kubernetes.io/projected/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-kube-api-access-p4gbx\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:56.749968 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:56.749973 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a62275-1d4e-4a65-84e0-ced4b6d3b39b-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:57.347601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:57.347566 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" Nov 02 20:42:57.347601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:57.347597 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6" event={"ID":"18a62275-1d4e-4a65-84e0-ced4b6d3b39b","Type":"ContainerDied","Data":"40104fe4ee1badfe0277d3bada6438d6957a541987e9b285871c1076cb2773ec"} Nov 02 20:42:57.347819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:57.347646 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40104fe4ee1badfe0277d3bada6438d6957a541987e9b285871c1076cb2773ec" Nov 02 20:42:57.507416 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:42:57.507383 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72161e39-06e8-4506-840a-701972593d9a" path="/var/lib/kubelet/pods/72161e39-06e8-4506-840a-701972593d9a/volumes" Nov 02 20:43:00.863642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:00.863599 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" podUID="4b0f654d-dc58-4977-a3a0-b7d35b280f7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:02.127708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:02.127667 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj"] Nov 02 20:43:02.128159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:02.127886 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" containerID="cri-o://470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab" gracePeriod=30 Nov 02 20:43:02.934660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:02.934617 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp"] Nov 02 20:43:02.935121 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:02.935107 5399 cpu_manager.go:414] "containerMap: removing container" podUID="18a62275-1d4e-4a65-84e0-ced4b6d3b39b" containerName="revoke-keys" Nov 02 20:43:02.935169 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:02.935122 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a62275-1d4e-4a65-84e0-ced4b6d3b39b" containerName="revoke-keys" Nov 02 20:43:02.935236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:02.935226 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a62275-1d4e-4a65-84e0-ced4b6d3b39b" containerName="revoke-keys" Nov 02 20:43:02.938768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:02.938753 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:02.990855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:02.990825 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp"] Nov 02 20:43:03.014727 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.014687 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gwr\" (UniqueName: \"kubernetes.io/projected/e572463d-165b-4df8-b5a9-9dd602ebeea0-kube-api-access-q5gwr\") pod \"maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:03.014944 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.014925 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:03.015017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.014975 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:03.033461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.033429 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:43:03.033461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.033457 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:43:03.033651 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.033457 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:43:03.033933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.033919 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:43:03.116230 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.116189 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:03.116472 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.116240 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:03.116472 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.116262 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gwr\" (UniqueName: \"kubernetes.io/projected/e572463d-165b-4df8-b5a9-9dd602ebeea0-kube-api-access-q5gwr\") pod \"maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:03.117159 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.117132 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:03.118708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.118686 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:03.169927 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.169894 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gwr\" (UniqueName: \"kubernetes.io/projected/e572463d-165b-4df8-b5a9-9dd602ebeea0-kube-api-access-q5gwr\") pod \"maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:03.252102 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.252020 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:03.619488 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.619453 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp"] Nov 02 20:43:03.620787 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:43:03.620745 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode572463d_165b_4df8_b5a9_9dd602ebeea0.slice/crio-19632ea3408a8914bd568b32c6335b7cf9a6ebd7d59da6aefa44794cf781fc00 WatchSource:0}: Error finding container 19632ea3408a8914bd568b32c6335b7cf9a6ebd7d59da6aefa44794cf781fc00: Status 404 returned error can't find the container with id 19632ea3408a8914bd568b32c6335b7cf9a6ebd7d59da6aefa44794cf781fc00 Nov 02 20:43:03.998905 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:03.998859 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl"] Nov 02 20:43:04.397916 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:04.397870 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" event={"ID":"e572463d-165b-4df8-b5a9-9dd602ebeea0","Type":"ContainerStarted","Data":"08ce85532becbc6a8623e0564c2efc30df91786c123abc0b8302a2b1cb0a5e57"} Nov 02 20:43:04.398456 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:04.397929 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" event={"ID":"e572463d-165b-4df8-b5a9-9dd602ebeea0","Type":"ContainerStarted","Data":"19632ea3408a8914bd568b32c6335b7cf9a6ebd7d59da6aefa44794cf781fc00"} Nov 02 20:43:04.505675 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:04.505627 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" podStartSLOduration=2.505612936 podStartE2EDuration="2.505612936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:04.504260674 +0000 UTC m=+3027.114912769" watchObservedRunningTime="2026-11-02 20:43:04.505612936 +0000 UTC m=+3027.116265027" Nov 02 20:43:04.867145 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:04.867120 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:43:04.932419 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:04.932386 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-plugins-config-volume\") pod \"4b0f654d-dc58-4977-a3a0-b7d35b280f7c\" (UID: \"4b0f654d-dc58-4977-a3a0-b7d35b280f7c\") " Nov 02 20:43:04.932612 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:04.932468 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-kube-api-access-v2rxw\" (UniqueName: \"kubernetes.io/projected/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-kube-api-access-v2rxw\") pod \"4b0f654d-dc58-4977-a3a0-b7d35b280f7c\" (UID: \"4b0f654d-dc58-4977-a3a0-b7d35b280f7c\") " Nov 02 20:43:04.932762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:04.932739 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-plugins-config-volume" pod "4b0f654d-dc58-4977-a3a0-b7d35b280f7c" (UID: "4b0f654d-dc58-4977-a3a0-b7d35b280f7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:04.934633 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:04.934608 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-kube-api-access-v2rxw" pod "4b0f654d-dc58-4977-a3a0-b7d35b280f7c" (UID: "4b0f654d-dc58-4977-a3a0-b7d35b280f7c"). InnerVolumeSpecName "kube-api-access-v2rxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:05.033405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.033319 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2rxw\" (UniqueName: \"kubernetes.io/projected/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-kube-api-access-v2rxw\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:05.033405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.033353 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0f654d-dc58-4977-a3a0-b7d35b280f7c-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:05.405343 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.405306 5399 generic.go:356] "Generic (PLEG): container finished" podID="4b0f654d-dc58-4977-a3a0-b7d35b280f7c" containerID="c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94" exitCode=0 Nov 02 20:43:05.405786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.405358 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" Nov 02 20:43:05.405786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.405393 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" event={"ID":"4b0f654d-dc58-4977-a3a0-b7d35b280f7c","Type":"ContainerDied","Data":"c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94"} Nov 02 20:43:05.405786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.405446 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4" event={"ID":"4b0f654d-dc58-4977-a3a0-b7d35b280f7c","Type":"ContainerDied","Data":"a808f8f2e4b77f75606c0e28a5c79d2c42edec2e0ac7e768c8526e230bae2fff"} Nov 02 20:43:05.405786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.405466 5399 scope.go:122] "RemoveContainer" containerID="c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94" Nov 02 20:43:05.407322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.407268 5399 generic.go:356] "Generic (PLEG): container finished" podID="e572463d-165b-4df8-b5a9-9dd602ebeea0" containerID="08ce85532becbc6a8623e0564c2efc30df91786c123abc0b8302a2b1cb0a5e57" exitCode=0 Nov 02 20:43:05.407443 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.407353 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" event={"ID":"e572463d-165b-4df8-b5a9-9dd602ebeea0","Type":"ContainerDied","Data":"08ce85532becbc6a8623e0564c2efc30df91786c123abc0b8302a2b1cb0a5e57"} Nov 02 20:43:05.416372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.416352 5399 scope.go:122] "RemoveContainer" containerID="c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94" Nov 02 20:43:05.416629 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:05.416611 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94\": container with ID starting with c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94 not found: ID does not exist" containerID="c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94" Nov 02 20:43:05.416681 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.416636 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94"} err="failed to get container status \"c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94\": rpc error: code = NotFound desc = could not find container \"c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94\": container with ID starting with c3fe4f3f70baa37072d2684fb6729cbe65d9927675894af9ab4a5840eed0ad94 not found: ID does not exist" Nov 02 20:43:05.558504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.558470 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4"] Nov 02 20:43:05.604351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.604314 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9ec0-7dc4d4cf87-t4bw4"] Nov 02 20:43:05.786405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.786324 5399 patch_prober.go:29] interesting pod/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:43:05.786405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:05.786378 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:43:06.563513 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:06.563484 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:06.650047 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:06.650014 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-openshift-ca\") pod \"e572463d-165b-4df8-b5a9-9dd602ebeea0\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " Nov 02 20:43:06.650242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:06.650145 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e572463d-165b-4df8-b5a9-9dd602ebeea0-kube-api-access-q5gwr\" (UniqueName: \"kubernetes.io/projected/e572463d-165b-4df8-b5a9-9dd602ebeea0-kube-api-access-q5gwr\") pod \"e572463d-165b-4df8-b5a9-9dd602ebeea0\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " Nov 02 20:43:06.650242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:06.650196 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-xks-ca\") pod \"e572463d-165b-4df8-b5a9-9dd602ebeea0\" (UID: \"e572463d-165b-4df8-b5a9-9dd602ebeea0\") " Nov 02 20:43:06.650492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:06.650467 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-openshift-ca" pod "e572463d-165b-4df8-b5a9-9dd602ebeea0" (UID: "e572463d-165b-4df8-b5a9-9dd602ebeea0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:06.650650 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:06.650626 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:06.652340 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:06.652317 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e572463d-165b-4df8-b5a9-9dd602ebeea0-kube-api-access-q5gwr" pod "e572463d-165b-4df8-b5a9-9dd602ebeea0" (UID: "e572463d-165b-4df8-b5a9-9dd602ebeea0"). InnerVolumeSpecName "kube-api-access-q5gwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:06.652433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:06.652394 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-xks-ca" pod "e572463d-165b-4df8-b5a9-9dd602ebeea0" (UID: "e572463d-165b-4df8-b5a9-9dd602ebeea0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:06.751405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:06.751320 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5gwr\" (UniqueName: \"kubernetes.io/projected/e572463d-165b-4df8-b5a9-9dd602ebeea0-kube-api-access-q5gwr\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:06.751405 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:06.751349 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e572463d-165b-4df8-b5a9-9dd602ebeea0-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.372831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.372804 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:43:07.423520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.423442 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" Nov 02 20:43:07.423520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.423471 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp" event={"ID":"e572463d-165b-4df8-b5a9-9dd602ebeea0","Type":"ContainerDied","Data":"19632ea3408a8914bd568b32c6335b7cf9a6ebd7d59da6aefa44794cf781fc00"} Nov 02 20:43:07.423741 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.423528 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19632ea3408a8914bd568b32c6335b7cf9a6ebd7d59da6aefa44794cf781fc00" Nov 02 20:43:07.425478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.425450 5399 generic.go:356] "Generic (PLEG): container finished" podID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerID="470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab" exitCode=0 Nov 02 20:43:07.425614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.425498 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" Nov 02 20:43:07.425614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.425515 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" event={"ID":"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f","Type":"ContainerDied","Data":"470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab"} Nov 02 20:43:07.425614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.425552 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj" event={"ID":"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f","Type":"ContainerDied","Data":"f1fbfb11da2be5ef0ff22ee127e92bc4a51921023b9192b44a8b39613d20a17b"} Nov 02 20:43:07.425614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.425574 5399 scope.go:122] "RemoveContainer" containerID="470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab" Nov 02 20:43:07.435839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.435824 5399 scope.go:122] "RemoveContainer" containerID="a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca" Nov 02 20:43:07.444900 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.444883 5399 scope.go:122] "RemoveContainer" containerID="470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab" Nov 02 20:43:07.445180 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:07.445163 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab\": container with ID starting with 470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab not found: ID does not exist" containerID="470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab" Nov 02 20:43:07.445228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.445190 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab"} err="failed to get container status \"470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab\": rpc error: code = NotFound desc = could not find container \"470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab\": container with ID starting with 470c0e8f9ee49fa6cdb1203fc53d96cc7d4913b2b4f5c839c8cba15790c75dab not found: ID does not exist" Nov 02 20:43:07.445228 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.445209 5399 scope.go:122] "RemoveContainer" containerID="a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca" Nov 02 20:43:07.445573 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:07.445555 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca\": container with ID starting with a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca not found: ID does not exist" containerID="a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca" Nov 02 20:43:07.445614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.445581 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca"} err="failed to get container status \"a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca\": rpc error: code = NotFound desc = could not find container \"a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca\": container with ID starting with a4ab153333da7766920c63e45c901bc6a39c26b53b5eae7086f8f84811fc60ca not found: ID does not exist" Nov 02 20:43:07.458220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458199 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-token\" (UniqueName: \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-token\") pod \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " Nov 02 20:43:07.458352 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458254 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-data\") pod \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " Nov 02 20:43:07.458401 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458351 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-certs\") pod \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " Nov 02 20:43:07.458436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458404 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-kube-api-access-zx5tk\" (UniqueName: \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-kube-api-access-zx5tk\") pod \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " Nov 02 20:43:07.458483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458436 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-credential-socket\") pod \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " Nov 02 20:43:07.458483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458461 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istiod-ca-cert\") pod \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " Nov 02 20:43:07.458590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458510 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-socket\") pod \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " Nov 02 20:43:07.458590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458545 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-data" pod "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" (UID: "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:07.458701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458586 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-envoy\") pod \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " Nov 02 20:43:07.458701 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458619 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-podinfo\") pod \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\" (UID: \"961e4fe5-5f3a-4d7f-981c-98fe9b42da4f\") " Nov 02 20:43:07.458819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458741 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-certs" pod "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" (UID: "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:07.458819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458753 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-credential-socket" pod "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" (UID: "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:07.458948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458840 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-socket" pod "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" (UID: "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:07.458948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.458873 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istiod-ca-cert" pod "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" (UID: "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:07.459162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.459144 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.459229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.459166 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.459229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.459179 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.459229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.459187 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.459229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.459196 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.460651 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.460630 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-envoy" pod "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" (UID: "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:07.460651 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.460641 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-kube-api-access-zx5tk" pod "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" (UID: "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f"). InnerVolumeSpecName "kube-api-access-zx5tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:07.460826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.460813 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-token" pod "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" (UID: "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:07.460908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.460896 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-podinfo" pod "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" (UID: "961e4fe5-5f3a-4d7f-981c-98fe9b42da4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:07.516476 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.516442 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b0f654d-dc58-4977-a3a0-b7d35b280f7c" path="/var/lib/kubelet/pods/4b0f654d-dc58-4977-a3a0-b7d35b280f7c/volumes" Nov 02 20:43:07.560871 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.560831 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.561003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.560874 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.561003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.560891 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.561003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.560906 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx5tk\" (UniqueName: \"kubernetes.io/projected/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f-kube-api-access-zx5tk\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:07.818067 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.818029 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:43:07.835283 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.835252 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj"] Nov 02 20:43:07.837964 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:07.837944 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bb78ba-openshift-default-5db8c47859-mhzvj"] Nov 02 20:43:08.999214 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:08.999160 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" podUID="3b1b1358-8da7-4ee0-afae-fa9af7e35144" containerName="payload-processing" containerID="cri-o://f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e" gracePeriod=25 Nov 02 20:43:09.507629 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:09.507596 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" path="/var/lib/kubelet/pods/961e4fe5-5f3a-4d7f-981c-98fe9b42da4f/volumes" Nov 02 20:43:11.211005 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:11.210962 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" podUID="3b1b1358-8da7-4ee0-afae-fa9af7e35144" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:13.958682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:13.958646 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6"] Nov 02 20:43:14.005020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:14.004985 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-aaa3e7-wdlh6"] Nov 02 20:43:15.507530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:15.507494 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a62275-1d4e-4a65-84e0-ced4b6d3b39b" path="/var/lib/kubelet/pods/18a62275-1d4e-4a65-84e0-ced4b6d3b39b/volumes" Nov 02 20:43:18.237823 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:18.237785 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:43:20.537123 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:20.537083 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh"] Nov 02 20:43:20.537537 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:20.537346 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" containerID="cri-o://f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce" gracePeriod=30 Nov 02 20:43:21.210551 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:21.210513 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" podUID="3b1b1358-8da7-4ee0-afae-fa9af7e35144" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:25.792041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.792014 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:43:25.865627 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.865599 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-envoy\") pod \"6debae9d-b9fb-40b8-9863-76e329edfdaa\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " Nov 02 20:43:25.865849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.865641 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-token\" (UniqueName: \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-token\") pod \"6debae9d-b9fb-40b8-9863-76e329edfdaa\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " Nov 02 20:43:25.865849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.865676 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-certs\") pod \"6debae9d-b9fb-40b8-9863-76e329edfdaa\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " Nov 02 20:43:25.865849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.865697 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-data\") pod \"6debae9d-b9fb-40b8-9863-76e329edfdaa\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " Nov 02 20:43:25.865849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.865745 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-kube-api-access-swkhs\" (UniqueName: \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-kube-api-access-swkhs\") pod \"6debae9d-b9fb-40b8-9863-76e329edfdaa\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " Nov 02 20:43:25.865849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.865801 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-podinfo\") pod \"6debae9d-b9fb-40b8-9863-76e329edfdaa\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " Nov 02 20:43:25.865849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.865849 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6debae9d-b9fb-40b8-9863-76e329edfdaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6debae9d-b9fb-40b8-9863-76e329edfdaa-istiod-ca-cert\") pod \"6debae9d-b9fb-40b8-9863-76e329edfdaa\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " Nov 02 20:43:25.866198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.865904 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-credential-socket\") pod \"6debae9d-b9fb-40b8-9863-76e329edfdaa\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " Nov 02 20:43:25.866198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.865934 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-socket\") pod \"6debae9d-b9fb-40b8-9863-76e329edfdaa\" (UID: \"6debae9d-b9fb-40b8-9863-76e329edfdaa\") " Nov 02 20:43:25.866198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.866005 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-data" pod "6debae9d-b9fb-40b8-9863-76e329edfdaa" (UID: "6debae9d-b9fb-40b8-9863-76e329edfdaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:25.866198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.866025 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-certs" pod "6debae9d-b9fb-40b8-9863-76e329edfdaa" (UID: "6debae9d-b9fb-40b8-9863-76e329edfdaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:25.866475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.866413 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:25.866475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.866436 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:25.866475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.866442 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-credential-socket" pod "6debae9d-b9fb-40b8-9863-76e329edfdaa" (UID: "6debae9d-b9fb-40b8-9863-76e329edfdaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:25.866475 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.866452 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6debae9d-b9fb-40b8-9863-76e329edfdaa-istiod-ca-cert" pod "6debae9d-b9fb-40b8-9863-76e329edfdaa" (UID: "6debae9d-b9fb-40b8-9863-76e329edfdaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:25.866661 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.866503 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-socket" pod "6debae9d-b9fb-40b8-9863-76e329edfdaa" (UID: "6debae9d-b9fb-40b8-9863-76e329edfdaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:25.868031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.868002 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-envoy" pod "6debae9d-b9fb-40b8-9863-76e329edfdaa" (UID: "6debae9d-b9fb-40b8-9863-76e329edfdaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:25.868216 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.868200 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-kube-api-access-swkhs" pod "6debae9d-b9fb-40b8-9863-76e329edfdaa" (UID: "6debae9d-b9fb-40b8-9863-76e329edfdaa"). InnerVolumeSpecName "kube-api-access-swkhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:25.868433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.868413 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-podinfo" pod "6debae9d-b9fb-40b8-9863-76e329edfdaa" (UID: "6debae9d-b9fb-40b8-9863-76e329edfdaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:25.868517 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.868443 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-token" pod "6debae9d-b9fb-40b8-9863-76e329edfdaa" (UID: "6debae9d-b9fb-40b8-9863-76e329edfdaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:25.967609 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.967575 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:25.967609 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.967607 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:25.967609 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.967617 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:25.967848 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.967626 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swkhs\" (UniqueName: \"kubernetes.io/projected/6debae9d-b9fb-40b8-9863-76e329edfdaa-kube-api-access-swkhs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:25.967848 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.967636 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6debae9d-b9fb-40b8-9863-76e329edfdaa-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:25.967848 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.967644 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6debae9d-b9fb-40b8-9863-76e329edfdaa-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:25.967848 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.967654 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6debae9d-b9fb-40b8-9863-76e329edfdaa-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:25.996784 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:25.996744 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q"] Nov 02 20:43:26.545994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.545960 5399 generic.go:356] "Generic (PLEG): container finished" podID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerID="f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce" exitCode=0 Nov 02 20:43:26.546178 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.546010 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" Nov 02 20:43:26.546178 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.546049 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" event={"ID":"6debae9d-b9fb-40b8-9863-76e329edfdaa","Type":"ContainerDied","Data":"f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce"} Nov 02 20:43:26.546178 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.546104 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh" event={"ID":"6debae9d-b9fb-40b8-9863-76e329edfdaa","Type":"ContainerDied","Data":"776d94e0ea11110c1ae85ef562f6987b82cf15ffc604e674ddf3f5468877ecf5"} Nov 02 20:43:26.546178 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.546128 5399 scope.go:122] "RemoveContainer" containerID="f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce" Nov 02 20:43:26.557115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.557097 5399 scope.go:122] "RemoveContainer" containerID="45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19" Nov 02 20:43:26.566055 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.566040 5399 scope.go:122] "RemoveContainer" containerID="f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce" Nov 02 20:43:26.566342 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:26.566319 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce\": container with ID starting with f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce not found: ID does not exist" containerID="f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce" Nov 02 20:43:26.566441 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.566350 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce"} err="failed to get container status \"f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce\": rpc error: code = NotFound desc = could not find container \"f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce\": container with ID starting with f972896c1b3685a964003b33896c342d717532b42522cab9c258ebd0e15602ce not found: ID does not exist" Nov 02 20:43:26.566441 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.566368 5399 scope.go:122] "RemoveContainer" containerID="45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19" Nov 02 20:43:26.566839 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:26.566817 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19\": container with ID starting with 45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19 not found: ID does not exist" containerID="45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19" Nov 02 20:43:26.566914 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.566846 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19"} err="failed to get container status \"45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19\": rpc error: code = NotFound desc = could not find container \"45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19\": container with ID starting with 45b1244a6ba57b7590122fa7698ba0b180c68df07a819eba48790d053f88ab19 not found: ID does not exist" Nov 02 20:43:26.699202 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.699162 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh"] Nov 02 20:43:26.745578 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:26.745538 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-aaa3e7-openshift-default-54b9fbf6b5-9rtgh"] Nov 02 20:43:27.507365 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:27.507335 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" path="/var/lib/kubelet/pods/6debae9d-b9fb-40b8-9863-76e329edfdaa/volumes" Nov 02 20:43:28.244997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:28.244964 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:43:29.254007 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.253980 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:43:29.401104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.401070 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1b1358-8da7-4ee0-afae-fa9af7e35144-kube-api-access-vtkcc\" (UniqueName: \"kubernetes.io/projected/3b1b1358-8da7-4ee0-afae-fa9af7e35144-kube-api-access-vtkcc\") pod \"3b1b1358-8da7-4ee0-afae-fa9af7e35144\" (UID: \"3b1b1358-8da7-4ee0-afae-fa9af7e35144\") " Nov 02 20:43:29.401342 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.401155 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1b1358-8da7-4ee0-afae-fa9af7e35144-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1b1358-8da7-4ee0-afae-fa9af7e35144-plugins-config-volume\") pod \"3b1b1358-8da7-4ee0-afae-fa9af7e35144\" (UID: \"3b1b1358-8da7-4ee0-afae-fa9af7e35144\") " Nov 02 20:43:29.401551 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.401527 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1b1358-8da7-4ee0-afae-fa9af7e35144-plugins-config-volume" pod "3b1b1358-8da7-4ee0-afae-fa9af7e35144" (UID: "3b1b1358-8da7-4ee0-afae-fa9af7e35144"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:29.403255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.403235 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1b1358-8da7-4ee0-afae-fa9af7e35144-kube-api-access-vtkcc" pod "3b1b1358-8da7-4ee0-afae-fa9af7e35144" (UID: "3b1b1358-8da7-4ee0-afae-fa9af7e35144"). InnerVolumeSpecName "kube-api-access-vtkcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:29.502197 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.502165 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtkcc\" (UniqueName: \"kubernetes.io/projected/3b1b1358-8da7-4ee0-afae-fa9af7e35144-kube-api-access-vtkcc\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:29.502197 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.502195 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1b1358-8da7-4ee0-afae-fa9af7e35144-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:29.565908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.565875 5399 generic.go:356] "Generic (PLEG): container finished" podID="3b1b1358-8da7-4ee0-afae-fa9af7e35144" containerID="f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e" exitCode=0 Nov 02 20:43:29.566050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.565933 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" Nov 02 20:43:29.566050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.565959 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" event={"ID":"3b1b1358-8da7-4ee0-afae-fa9af7e35144","Type":"ContainerDied","Data":"f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e"} Nov 02 20:43:29.566050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.566013 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl" event={"ID":"3b1b1358-8da7-4ee0-afae-fa9af7e35144","Type":"ContainerDied","Data":"0b0faa66b1a7966b80eab79d133bbbb194e0b232d2f4b582b61341a7e04b0fe6"} Nov 02 20:43:29.566050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.566038 5399 scope.go:122] "RemoveContainer" containerID="f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e" Nov 02 20:43:29.576076 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.576055 5399 scope.go:122] "RemoveContainer" containerID="f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e" Nov 02 20:43:29.576385 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:29.576361 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e\": container with ID starting with f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e not found: ID does not exist" containerID="f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e" Nov 02 20:43:29.576530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.576389 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e"} err="failed to get container status \"f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e\": rpc error: code = NotFound desc = could not find container \"f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e\": container with ID starting with f08f848fe8b565f5332c9b25c844026aa21fad6252ca9664a84f578d5db3e67e not found: ID does not exist" Nov 02 20:43:29.717948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.717852 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl"] Nov 02 20:43:29.720356 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:29.720331 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aaa3e7-f85d4dcff-fwwjl"] Nov 02 20:43:31.506764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:31.506724 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1b1358-8da7-4ee0-afae-fa9af7e35144" path="/var/lib/kubelet/pods/3b1b1358-8da7-4ee0-afae-fa9af7e35144/volumes" Nov 02 20:43:34.020940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:34.020903 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp"] Nov 02 20:43:34.067218 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:34.067183 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2b3b9e-g5tnp"] Nov 02 20:43:35.507561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:35.507524 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e572463d-165b-4df8-b5a9-9dd602ebeea0" path="/var/lib/kubelet/pods/e572463d-165b-4df8-b5a9-9dd602ebeea0/volumes" Nov 02 20:43:38.034598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.034566 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss"] Nov 02 20:43:38.035042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.034989 5399 cpu_manager.go:414] "containerMap: removing container" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" Nov 02 20:43:38.035042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035002 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" Nov 02 20:43:38.035042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035010 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e572463d-165b-4df8-b5a9-9dd602ebeea0" containerName="revoke-keys" Nov 02 20:43:38.035042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035016 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e572463d-165b-4df8-b5a9-9dd602ebeea0" containerName="revoke-keys" Nov 02 20:43:38.035042 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035045 5399 cpu_manager.go:414] "containerMap: removing container" podUID="4b0f654d-dc58-4977-a3a0-b7d35b280f7c" containerName="payload-pre-processing" Nov 02 20:43:38.035311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035050 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b0f654d-dc58-4977-a3a0-b7d35b280f7c" containerName="payload-pre-processing" Nov 02 20:43:38.035311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035062 5399 cpu_manager.go:414] "containerMap: removing container" podUID="3b1b1358-8da7-4ee0-afae-fa9af7e35144" containerName="payload-processing" Nov 02 20:43:38.035311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035068 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1b1358-8da7-4ee0-afae-fa9af7e35144" containerName="payload-processing" Nov 02 20:43:38.035311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035082 5399 cpu_manager.go:414] "containerMap: removing container" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" Nov 02 20:43:38.035311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035087 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" Nov 02 20:43:38.035311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035175 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" Nov 02 20:43:38.035311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035188 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" Nov 02 20:43:38.035311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035197 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e572463d-165b-4df8-b5a9-9dd602ebeea0" containerName="revoke-keys" Nov 02 20:43:38.035311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035212 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b0f654d-dc58-4977-a3a0-b7d35b280f7c" containerName="payload-pre-processing" Nov 02 20:43:38.035311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.035221 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1b1358-8da7-4ee0-afae-fa9af7e35144" containerName="payload-processing" Nov 02 20:43:38.038577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.038553 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.085960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.085925 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e87257-dxhss\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.086142 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.086022 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e87257-dxhss\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.086142 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.086051 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sckhz\" (UniqueName: \"kubernetes.io/projected/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-kube-api-access-sckhz\") pod \"maas-api-revoke-sub-28e87257-dxhss\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.090574 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.090543 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss"] Nov 02 20:43:38.133839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.133812 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:43:38.133839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.133811 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:43:38.136707 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.136688 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:43:38.136837 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.136819 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:43:38.186812 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.186774 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e87257-dxhss\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.186988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.186858 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e87257-dxhss\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.186988 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.186888 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sckhz\" (UniqueName: \"kubernetes.io/projected/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-kube-api-access-sckhz\") pod \"maas-api-revoke-sub-28e87257-dxhss\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.187542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.187519 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e87257-dxhss\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.189008 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.188989 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e87257-dxhss\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.240839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.240812 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sckhz\" (UniqueName: \"kubernetes.io/projected/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-kube-api-access-sckhz\") pod \"maas-api-revoke-sub-28e87257-dxhss\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.353081 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.353048 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:38.717669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:38.717643 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss"] Nov 02 20:43:38.719317 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:43:38.719263 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2421f9a_57bf_49e9_90b5_6ac6b35a503b.slice/crio-c02e07e759fe1c9f4020f6f68b36d74aae122d69158632c02fe379786163b26a WatchSource:0}: Error finding container c02e07e759fe1c9f4020f6f68b36d74aae122d69158632c02fe379786163b26a: Status 404 returned error can't find the container with id c02e07e759fe1c9f4020f6f68b36d74aae122d69158632c02fe379786163b26a Nov 02 20:43:39.315575 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:39.315543 5399 scope.go:122] "RemoveContainer" containerID="ff02f27b39611ec692051a977f3e36c8821d77a31bf3347faaf7bce6105598c3" Nov 02 20:43:39.325046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:39.325025 5399 scope.go:122] "RemoveContainer" containerID="c697e658e41d92158478c076e54c8ad73836367a25fa0f20d90a7dca0079a1ca" Nov 02 20:43:39.637592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:39.637552 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" event={"ID":"a2421f9a-57bf-49e9-90b5-6ac6b35a503b","Type":"ContainerStarted","Data":"9ca0148cde2100f47281a3ff3d4c66ac42bad537946aaa44d3af89eab8a75da2"} Nov 02 20:43:39.637750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:39.637632 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" event={"ID":"a2421f9a-57bf-49e9-90b5-6ac6b35a503b","Type":"ContainerStarted","Data":"c02e07e759fe1c9f4020f6f68b36d74aae122d69158632c02fe379786163b26a"} Nov 02 20:43:40.645020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:40.644986 5399 generic.go:356] "Generic (PLEG): container finished" podID="a2421f9a-57bf-49e9-90b5-6ac6b35a503b" containerID="9ca0148cde2100f47281a3ff3d4c66ac42bad537946aaa44d3af89eab8a75da2" exitCode=0 Nov 02 20:43:40.645479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:40.645075 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" event={"ID":"a2421f9a-57bf-49e9-90b5-6ac6b35a503b","Type":"ContainerDied","Data":"9ca0148cde2100f47281a3ff3d4c66ac42bad537946aaa44d3af89eab8a75da2"} Nov 02 20:43:41.017017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.016927 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" containerID="cri-o://ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f" gracePeriod=45 Nov 02 20:43:41.277781 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.277719 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:43:41.310652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.310622 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tls-certs\") pod \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " Nov 02 20:43:41.310794 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.310676 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-home\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-home\") pod \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " Nov 02 20:43:41.310794 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.310695 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-model-cache\") pod \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " Nov 02 20:43:41.310874 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.310796 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-kube-api-access-5krnk\" (UniqueName: \"kubernetes.io/projected/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-kube-api-access-5krnk\") pod \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " Nov 02 20:43:41.310874 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.310866 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-dshm\") pod \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " Nov 02 20:43:41.310967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.310913 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tmp-dir\") pod \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\" (UID: \"d0e04ae7-2c61-4595-8e07-2a7349bb01b9\") " Nov 02 20:43:41.310967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.310931 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-model-cache" pod "d0e04ae7-2c61-4595-8e07-2a7349bb01b9" (UID: "d0e04ae7-2c61-4595-8e07-2a7349bb01b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.310967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.310947 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-home" pod "d0e04ae7-2c61-4595-8e07-2a7349bb01b9" (UID: "d0e04ae7-2c61-4595-8e07-2a7349bb01b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.311213 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.311194 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tmp-dir" pod "d0e04ae7-2c61-4595-8e07-2a7349bb01b9" (UID: "d0e04ae7-2c61-4595-8e07-2a7349bb01b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.311313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.311297 5399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-home\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.311385 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.311319 5399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-model-cache\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.311385 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.311328 5399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tmp-dir\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.313003 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.312971 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-kube-api-access-5krnk" pod "d0e04ae7-2c61-4595-8e07-2a7349bb01b9" (UID: "d0e04ae7-2c61-4595-8e07-2a7349bb01b9"). InnerVolumeSpecName "kube-api-access-5krnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:41.313117 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.313024 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tls-certs" pod "d0e04ae7-2c61-4595-8e07-2a7349bb01b9" (UID: "d0e04ae7-2c61-4595-8e07-2a7349bb01b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:41.313117 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.313063 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-dshm" pod "d0e04ae7-2c61-4595-8e07-2a7349bb01b9" (UID: "d0e04ae7-2c61-4595-8e07-2a7349bb01b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:41.411998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.411965 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5krnk\" (UniqueName: \"kubernetes.io/projected/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-kube-api-access-5krnk\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.411998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.411995 5399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-dshm\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.411998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.412004 5399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e04ae7-2c61-4595-8e07-2a7349bb01b9-tls-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.654844 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.654800 5399 generic.go:356] "Generic (PLEG): container finished" podID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerID="ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f" exitCode=0 Nov 02 20:43:41.655371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.654850 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" Nov 02 20:43:41.655371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.654893 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" event={"ID":"d0e04ae7-2c61-4595-8e07-2a7349bb01b9","Type":"ContainerDied","Data":"ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f"} Nov 02 20:43:41.655371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.654946 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q" event={"ID":"d0e04ae7-2c61-4595-8e07-2a7349bb01b9","Type":"ContainerDied","Data":"2553887b1790328cd1a7e537f73f24c733b5bb3d47a5e354e33d1756acf368e2"} Nov 02 20:43:41.655371 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.654969 5399 scope.go:122] "RemoveContainer" containerID="ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f" Nov 02 20:43:41.667054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.667024 5399 scope.go:122] "RemoveContainer" containerID="ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f" Nov 02 20:43:41.667392 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:41.667371 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f\": container with ID starting with ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f not found: ID does not exist" containerID="ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f" Nov 02 20:43:41.667487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.667400 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f"} err="failed to get container status \"ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f\": rpc error: code = NotFound desc = could not find container \"ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f\": container with ID starting with ad111332d5a50e71bba21c156f83709de0e50d551670d03c1d652c374a544b9f not found: ID does not exist" Nov 02 20:43:41.783246 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.783218 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:41.809896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.809861 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q"] Nov 02 20:43:41.814566 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.814541 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-xks-ca\") pod \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " Nov 02 20:43:41.814632 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.814591 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-kube-api-access-sckhz\" (UniqueName: \"kubernetes.io/projected/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-kube-api-access-sckhz\") pod \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " Nov 02 20:43:41.814742 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.814727 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-openshift-ca\") pod \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\" (UID: \"a2421f9a-57bf-49e9-90b5-6ac6b35a503b\") " Nov 02 20:43:41.815077 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.815051 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-openshift-ca" pod "a2421f9a-57bf-49e9-90b5-6ac6b35a503b" (UID: "a2421f9a-57bf-49e9-90b5-6ac6b35a503b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:41.816635 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.816615 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-xks-ca" pod "a2421f9a-57bf-49e9-90b5-6ac6b35a503b" (UID: "a2421f9a-57bf-49e9-90b5-6ac6b35a503b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:41.816708 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.816681 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-kube-api-access-sckhz" pod "a2421f9a-57bf-49e9-90b5-6ac6b35a503b" (UID: "a2421f9a-57bf-49e9-90b5-6ac6b35a503b"). InnerVolumeSpecName "kube-api-access-sckhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:41.855697 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.855666 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc64d3/distinct-2-w4-cc64d3-kserve-7fb76c6d47-xl86q"] Nov 02 20:43:41.916114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.916030 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.916114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.916065 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:41.916114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:41.916082 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sckhz\" (UniqueName: \"kubernetes.io/projected/a2421f9a-57bf-49e9-90b5-6ac6b35a503b-kube-api-access-sckhz\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:42.662229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:42.662199 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" Nov 02 20:43:42.662229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:42.662219 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss" event={"ID":"a2421f9a-57bf-49e9-90b5-6ac6b35a503b","Type":"ContainerDied","Data":"c02e07e759fe1c9f4020f6f68b36d74aae122d69158632c02fe379786163b26a"} Nov 02 20:43:42.662776 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:42.662265 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c02e07e759fe1c9f4020f6f68b36d74aae122d69158632c02fe379786163b26a" Nov 02 20:43:43.506967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:43.506927 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" path="/var/lib/kubelet/pods/d0e04ae7-2c61-4595-8e07-2a7349bb01b9/volumes" Nov 02 20:43:45.908512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:45.908475 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4"] Nov 02 20:43:45.951509 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:45.951471 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk"] Nov 02 20:43:45.954965 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:45.954937 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk"] Nov 02 20:43:47.665313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:47.665215 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:43:48.236832 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:48.236787 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:43:52.972638 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:52.972600 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc"] Nov 02 20:43:53.085062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:53.085022 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc"] Nov 02 20:43:57.023902 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.023866 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm"] Nov 02 20:43:57.024318 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024310 5399 cpu_manager.go:414] "containerMap: removing container" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" Nov 02 20:43:57.024369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024321 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" Nov 02 20:43:57.024369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024340 5399 cpu_manager.go:414] "containerMap: removing container" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" Nov 02 20:43:57.024369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024347 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" Nov 02 20:43:57.024369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024360 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" Nov 02 20:43:57.024369 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024365 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" Nov 02 20:43:57.024534 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024372 5399 cpu_manager.go:414] "containerMap: removing container" podUID="a2421f9a-57bf-49e9-90b5-6ac6b35a503b" containerName="revoke-keys" Nov 02 20:43:57.024534 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024377 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2421f9a-57bf-49e9-90b5-6ac6b35a503b" containerName="revoke-keys" Nov 02 20:43:57.024534 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024444 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="961e4fe5-5f3a-4d7f-981c-98fe9b42da4f" containerName="istio-proxy" Nov 02 20:43:57.024534 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024453 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6debae9d-b9fb-40b8-9863-76e329edfdaa" containerName="istio-proxy" Nov 02 20:43:57.024534 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024460 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e04ae7-2c61-4595-8e07-2a7349bb01b9" containerName="main" Nov 02 20:43:57.024534 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.024466 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2421f9a-57bf-49e9-90b5-6ac6b35a503b" containerName="revoke-keys" Nov 02 20:43:57.029208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.029182 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.077886 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.077846 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ff4d7d0-br7nm\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.078055 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.077914 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9kb\" (UniqueName: \"kubernetes.io/projected/f04cadfc-8aff-42f7-b415-8fe0aa34a767-kube-api-access-6f9kb\") pod \"maas-api-revoke-sub-6ff4d7d0-br7nm\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.078224 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.078197 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ff4d7d0-br7nm\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.080864 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.080798 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm"] Nov 02 20:43:57.123966 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.123934 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:43:57.124155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.124021 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:43:57.124155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.124072 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:43:57.124155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.124141 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:43:57.178997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.178955 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ff4d7d0-br7nm\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.179194 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.179020 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ff4d7d0-br7nm\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.179194 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.179067 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9kb\" (UniqueName: \"kubernetes.io/projected/f04cadfc-8aff-42f7-b415-8fe0aa34a767-kube-api-access-6f9kb\") pod \"maas-api-revoke-sub-6ff4d7d0-br7nm\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.179935 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.179909 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ff4d7d0-br7nm\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.181780 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.181757 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ff4d7d0-br7nm\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.232673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.232635 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9kb\" (UniqueName: \"kubernetes.io/projected/f04cadfc-8aff-42f7-b415-8fe0aa34a767-kube-api-access-6f9kb\") pod \"maas-api-revoke-sub-6ff4d7d0-br7nm\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.352490 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.352452 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:43:57.720060 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.720030 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm"] Nov 02 20:43:57.721075 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:43:57.721046 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04cadfc_8aff_42f7_b415_8fe0aa34a767.slice/crio-9e025e01dd34990a7efac59ae4a6253f450211e802cfb006dc2234e9cf823f9c WatchSource:0}: Error finding container 9e025e01dd34990a7efac59ae4a6253f450211e802cfb006dc2234e9cf823f9c: Status 404 returned error can't find the container with id 9e025e01dd34990a7efac59ae4a6253f450211e802cfb006dc2234e9cf823f9c Nov 02 20:43:57.756047 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.756020 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" event={"ID":"f04cadfc-8aff-42f7-b415-8fe0aa34a767","Type":"ContainerStarted","Data":"9e025e01dd34990a7efac59ae4a6253f450211e802cfb006dc2234e9cf823f9c"} Nov 02 20:43:57.973222 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:57.973136 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" podUID="20aa971f-7e5c-4267-a7a8-b07b99676807" containerName="payload-processing" containerID="cri-o://20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27" gracePeriod=25 Nov 02 20:43:58.085895 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.085855 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" podUID="90935866-4032-47ca-8938-c965d81fdda0" containerName="payload-pre-processing" containerID="cri-o://07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04" gracePeriod=25 Nov 02 20:43:58.238017 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.237931 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:43:58.391542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.391508 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20aa971f-7e5c-4267-a7a8-b07b99676807-kube-api-access-8nwmf\" (UniqueName: \"kubernetes.io/projected/20aa971f-7e5c-4267-a7a8-b07b99676807-kube-api-access-8nwmf\") pod \"20aa971f-7e5c-4267-a7a8-b07b99676807\" (UID: \"20aa971f-7e5c-4267-a7a8-b07b99676807\") " Nov 02 20:43:58.391737 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.391646 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20aa971f-7e5c-4267-a7a8-b07b99676807-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20aa971f-7e5c-4267-a7a8-b07b99676807-plugins-config-volume\") pod \"20aa971f-7e5c-4267-a7a8-b07b99676807\" (UID: \"20aa971f-7e5c-4267-a7a8-b07b99676807\") " Nov 02 20:43:58.392112 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.392085 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20aa971f-7e5c-4267-a7a8-b07b99676807-plugins-config-volume" pod "20aa971f-7e5c-4267-a7a8-b07b99676807" (UID: "20aa971f-7e5c-4267-a7a8-b07b99676807"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:58.394830 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.394802 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20aa971f-7e5c-4267-a7a8-b07b99676807-kube-api-access-8nwmf" pod "20aa971f-7e5c-4267-a7a8-b07b99676807" (UID: "20aa971f-7e5c-4267-a7a8-b07b99676807"). InnerVolumeSpecName "kube-api-access-8nwmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:58.422350 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.422321 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:43:58.492792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.492709 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90935866-4032-47ca-8938-c965d81fdda0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90935866-4032-47ca-8938-c965d81fdda0-plugins-config-volume\") pod \"90935866-4032-47ca-8938-c965d81fdda0\" (UID: \"90935866-4032-47ca-8938-c965d81fdda0\") " Nov 02 20:43:58.492792 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.492772 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90935866-4032-47ca-8938-c965d81fdda0-kube-api-access-q69jw\" (UniqueName: \"kubernetes.io/projected/90935866-4032-47ca-8938-c965d81fdda0-kube-api-access-q69jw\") pod \"90935866-4032-47ca-8938-c965d81fdda0\" (UID: \"90935866-4032-47ca-8938-c965d81fdda0\") " Nov 02 20:43:58.493087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.493072 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20aa971f-7e5c-4267-a7a8-b07b99676807-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.493158 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.493088 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nwmf\" (UniqueName: \"kubernetes.io/projected/20aa971f-7e5c-4267-a7a8-b07b99676807-kube-api-access-8nwmf\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.493158 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.493139 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90935866-4032-47ca-8938-c965d81fdda0-plugins-config-volume" pod "90935866-4032-47ca-8938-c965d81fdda0" (UID: "90935866-4032-47ca-8938-c965d81fdda0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:58.495012 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.494978 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90935866-4032-47ca-8938-c965d81fdda0-kube-api-access-q69jw" pod "90935866-4032-47ca-8938-c965d81fdda0" (UID: "90935866-4032-47ca-8938-c965d81fdda0"). InnerVolumeSpecName "kube-api-access-q69jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:58.594187 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.594147 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q69jw\" (UniqueName: \"kubernetes.io/projected/90935866-4032-47ca-8938-c965d81fdda0-kube-api-access-q69jw\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.594187 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.594180 5399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90935866-4032-47ca-8938-c965d81fdda0-plugins-config-volume\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:58.765486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.765348 5399 generic.go:356] "Generic (PLEG): container finished" podID="f04cadfc-8aff-42f7-b415-8fe0aa34a767" containerID="b2dc56410f129134745859976f26ec4d98b5ff111d241b0905b8147aebd07eae" exitCode=0 Nov 02 20:43:58.765486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.765436 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" event={"ID":"f04cadfc-8aff-42f7-b415-8fe0aa34a767","Type":"ContainerDied","Data":"b2dc56410f129134745859976f26ec4d98b5ff111d241b0905b8147aebd07eae"} Nov 02 20:43:58.767249 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.767216 5399 generic.go:356] "Generic (PLEG): container finished" podID="90935866-4032-47ca-8938-c965d81fdda0" containerID="07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04" exitCode=0 Nov 02 20:43:58.767478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.767251 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" Nov 02 20:43:58.767478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.767295 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" event={"ID":"90935866-4032-47ca-8938-c965d81fdda0","Type":"ContainerDied","Data":"07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04"} Nov 02 20:43:58.767478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.767344 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc" event={"ID":"90935866-4032-47ca-8938-c965d81fdda0","Type":"ContainerDied","Data":"d2b97eb520fb221b8866e55a0334b39ab297ffd96a8118822437f6f5552366a9"} Nov 02 20:43:58.767478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.767374 5399 scope.go:122] "RemoveContainer" containerID="07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04" Nov 02 20:43:58.770436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.770410 5399 generic.go:356] "Generic (PLEG): container finished" podID="20aa971f-7e5c-4267-a7a8-b07b99676807" containerID="20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27" exitCode=0 Nov 02 20:43:58.770570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.770469 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" Nov 02 20:43:58.770570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.770520 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" event={"ID":"20aa971f-7e5c-4267-a7a8-b07b99676807","Type":"ContainerDied","Data":"20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27"} Nov 02 20:43:58.770570 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.770560 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc" event={"ID":"20aa971f-7e5c-4267-a7a8-b07b99676807","Type":"ContainerDied","Data":"ef46fc9712a16efab26b0513b84cc381c2731f231ebeb2a4629b970ae05e9cd1"} Nov 02 20:43:58.781776 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.781757 5399 scope.go:122] "RemoveContainer" containerID="07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04" Nov 02 20:43:58.782459 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:58.782436 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04\": container with ID starting with 07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04 not found: ID does not exist" containerID="07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04" Nov 02 20:43:58.782557 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.782471 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04"} err="failed to get container status \"07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04\": rpc error: code = NotFound desc = could not find container \"07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04\": container with ID starting with 07768d619f5fddc596adc4cba39e73390543ef4e739b206a522997ab8ae6fa04 not found: ID does not exist" Nov 02 20:43:58.782557 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.782494 5399 scope.go:122] "RemoveContainer" containerID="20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27" Nov 02 20:43:58.793922 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.793901 5399 scope.go:122] "RemoveContainer" containerID="20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27" Nov 02 20:43:58.794188 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:43:58.794169 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27\": container with ID starting with 20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27 not found: ID does not exist" containerID="20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27" Nov 02 20:43:58.794236 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.794195 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27"} err="failed to get container status \"20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27\": rpc error: code = NotFound desc = could not find container \"20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27\": container with ID starting with 20c753ea0c8b6644b70fa6d4ffb3f4c74742e88bfb4296ec727d64a62e137d27 not found: ID does not exist" Nov 02 20:43:58.979146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.979115 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc"] Nov 02 20:43:58.983239 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:58.983205 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-035e47-cf4fcc7d7-wjzkc"] Nov 02 20:43:59.082059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:59.082025 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc"] Nov 02 20:43:59.090194 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:59.090162 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-035e47-744944c75-phblc"] Nov 02 20:43:59.506920 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:59.506828 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20aa971f-7e5c-4267-a7a8-b07b99676807" path="/var/lib/kubelet/pods/20aa971f-7e5c-4267-a7a8-b07b99676807/volumes" Nov 02 20:43:59.507355 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:59.507334 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90935866-4032-47ca-8938-c965d81fdda0" path="/var/lib/kubelet/pods/90935866-4032-47ca-8938-c965d81fdda0/volumes" Nov 02 20:43:59.918562 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:43:59.918528 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:44:00.008380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.008344 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04cadfc-8aff-42f7-b415-8fe0aa34a767-kube-api-access-6f9kb\" (UniqueName: \"kubernetes.io/projected/f04cadfc-8aff-42f7-b415-8fe0aa34a767-kube-api-access-6f9kb\") pod \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " Nov 02 20:44:00.008562 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.008479 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-xks-ca\") pod \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " Nov 02 20:44:00.008562 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.008532 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-openshift-ca\") pod \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\" (UID: \"f04cadfc-8aff-42f7-b415-8fe0aa34a767\") " Nov 02 20:44:00.008933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.008903 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-openshift-ca" pod "f04cadfc-8aff-42f7-b415-8fe0aa34a767" (UID: "f04cadfc-8aff-42f7-b415-8fe0aa34a767"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:00.010646 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.010620 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-xks-ca" pod "f04cadfc-8aff-42f7-b415-8fe0aa34a767" (UID: "f04cadfc-8aff-42f7-b415-8fe0aa34a767"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:00.010740 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.010725 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04cadfc-8aff-42f7-b415-8fe0aa34a767-kube-api-access-6f9kb" pod "f04cadfc-8aff-42f7-b415-8fe0aa34a767" (UID: "f04cadfc-8aff-42f7-b415-8fe0aa34a767"). InnerVolumeSpecName "kube-api-access-6f9kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:00.110457 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.110417 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:00.110457 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.110447 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f04cadfc-8aff-42f7-b415-8fe0aa34a767-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:00.110457 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.110457 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f9kb\" (UniqueName: \"kubernetes.io/projected/f04cadfc-8aff-42f7-b415-8fe0aa34a767-kube-api-access-6f9kb\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:00.786863 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.786831 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" Nov 02 20:44:00.787045 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.786869 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm" event={"ID":"f04cadfc-8aff-42f7-b415-8fe0aa34a767","Type":"ContainerDied","Data":"9e025e01dd34990a7efac59ae4a6253f450211e802cfb006dc2234e9cf823f9c"} Nov 02 20:44:00.787045 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.786916 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e025e01dd34990a7efac59ae4a6253f450211e802cfb006dc2234e9cf823f9c" Nov 02 20:44:00.928601 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.928560 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" containerID="cri-o://8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a" gracePeriod=45 Nov 02 20:44:00.976129 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.976094 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" containerID="cri-o://973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55" gracePeriod=45 Nov 02 20:44:00.976312 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:00.976094 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" containerID="cri-o://5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f" gracePeriod=45 Nov 02 20:44:01.147993 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.147934 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused" Nov 02 20:44:01.164663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.164573 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": dial tcp 10.133.0.57:8000: connect: connection refused" Nov 02 20:44:01.249104 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.249074 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:44:01.277153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.277129 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:44:01.281508 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.281484 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:44:01.322797 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.322767 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead97984-d504-497d-b1ca-c2781eace145-kube-api-access-tpsb2\" (UniqueName: \"kubernetes.io/projected/ead97984-d504-497d-b1ca-c2781eace145-kube-api-access-tpsb2\") pod \"ead97984-d504-497d-b1ca-c2781eace145\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " Nov 02 20:44:01.322991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.322824 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-tmp-dir\") pod \"ead97984-d504-497d-b1ca-c2781eace145\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " Nov 02 20:44:01.322991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.322880 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-dshm\") pod \"ead97984-d504-497d-b1ca-c2781eace145\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " Nov 02 20:44:01.322991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.322900 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-model-cache\") pod \"ead97984-d504-497d-b1ca-c2781eace145\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " Nov 02 20:44:01.322991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.322924 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-home\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-home\") pod \"ead97984-d504-497d-b1ca-c2781eace145\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " Nov 02 20:44:01.322991 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.322952 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead97984-d504-497d-b1ca-c2781eace145-tls-certs\" (UniqueName: \"kubernetes.io/secret/ead97984-d504-497d-b1ca-c2781eace145-tls-certs\") pod \"ead97984-d504-497d-b1ca-c2781eace145\" (UID: \"ead97984-d504-497d-b1ca-c2781eace145\") " Nov 02 20:44:01.323314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.323178 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-tmp-dir" pod "ead97984-d504-497d-b1ca-c2781eace145" (UID: "ead97984-d504-497d-b1ca-c2781eace145"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.323314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.323228 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-model-cache" pod "ead97984-d504-497d-b1ca-c2781eace145" (UID: "ead97984-d504-497d-b1ca-c2781eace145"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.323434 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.323384 5399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-model-cache\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.323494 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.323432 5399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-tmp-dir\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.323563 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.323519 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-home" pod "ead97984-d504-497d-b1ca-c2781eace145" (UID: "ead97984-d504-497d-b1ca-c2781eace145"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.325179 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.325156 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead97984-d504-497d-b1ca-c2781eace145-kube-api-access-tpsb2" pod "ead97984-d504-497d-b1ca-c2781eace145" (UID: "ead97984-d504-497d-b1ca-c2781eace145"). InnerVolumeSpecName "kube-api-access-tpsb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:01.325265 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.325176 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead97984-d504-497d-b1ca-c2781eace145-tls-certs" pod "ead97984-d504-497d-b1ca-c2781eace145" (UID: "ead97984-d504-497d-b1ca-c2781eace145"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:01.325337 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.325260 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-dshm" pod "ead97984-d504-497d-b1ca-c2781eace145" (UID: "ead97984-d504-497d-b1ca-c2781eace145"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.424201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424165 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-home\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-home\") pod \"765d761d-8567-46a2-a441-c185a232f060\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " Nov 02 20:44:01.424201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424204 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-dshm\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-dshm\") pod \"765d761d-8567-46a2-a441-c185a232f060\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " Nov 02 20:44:01.424498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424234 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d267cc3b-bbd7-42aa-b457-504072df67db-kube-api-access-rrgqx\" (UniqueName: \"kubernetes.io/projected/d267cc3b-bbd7-42aa-b457-504072df67db-kube-api-access-rrgqx\") pod \"d267cc3b-bbd7-42aa-b457-504072df67db\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " Nov 02 20:44:01.424498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424294 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/765d761d-8567-46a2-a441-c185a232f060-tls-certs\" (UniqueName: \"kubernetes.io/secret/765d761d-8567-46a2-a441-c185a232f060-tls-certs\") pod \"765d761d-8567-46a2-a441-c185a232f060\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " Nov 02 20:44:01.424498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424347 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-model-cache\") pod \"765d761d-8567-46a2-a441-c185a232f060\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " Nov 02 20:44:01.424498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424394 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-dshm\") pod \"d267cc3b-bbd7-42aa-b457-504072df67db\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " Nov 02 20:44:01.424498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424425 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-home\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-home\") pod \"d267cc3b-bbd7-42aa-b457-504072df67db\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " Nov 02 20:44:01.424498 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424478 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-home" pod "765d761d-8567-46a2-a441-c185a232f060" (UID: "765d761d-8567-46a2-a441-c185a232f060"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.424893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424511 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-tmp-dir\") pod \"765d761d-8567-46a2-a441-c185a232f060\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " Nov 02 20:44:01.424893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424549 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d267cc3b-bbd7-42aa-b457-504072df67db-tls-certs\" (UniqueName: \"kubernetes.io/secret/d267cc3b-bbd7-42aa-b457-504072df67db-tls-certs\") pod \"d267cc3b-bbd7-42aa-b457-504072df67db\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " Nov 02 20:44:01.424893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424585 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765d761d-8567-46a2-a441-c185a232f060-kube-api-access-crvlq\" (UniqueName: \"kubernetes.io/projected/765d761d-8567-46a2-a441-c185a232f060-kube-api-access-crvlq\") pod \"765d761d-8567-46a2-a441-c185a232f060\" (UID: \"765d761d-8567-46a2-a441-c185a232f060\") " Nov 02 20:44:01.424893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424623 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-tmp-dir\") pod \"d267cc3b-bbd7-42aa-b457-504072df67db\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " Nov 02 20:44:01.424893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424670 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-model-cache" pod "765d761d-8567-46a2-a441-c185a232f060" (UID: "765d761d-8567-46a2-a441-c185a232f060"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.424893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424679 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-model-cache\") pod \"d267cc3b-bbd7-42aa-b457-504072df67db\" (UID: \"d267cc3b-bbd7-42aa-b457-504072df67db\") " Nov 02 20:44:01.424893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424704 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-home" pod "d267cc3b-bbd7-42aa-b457-504072df67db" (UID: "d267cc3b-bbd7-42aa-b457-504072df67db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.424893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424779 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-tmp-dir" pod "765d761d-8567-46a2-a441-c185a232f060" (UID: "765d761d-8567-46a2-a441-c185a232f060"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.424893 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424872 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-model-cache" pod "d267cc3b-bbd7-42aa-b457-504072df67db" (UID: "d267cc3b-bbd7-42aa-b457-504072df67db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.424994 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-tmp-dir" pod "d267cc3b-bbd7-42aa-b457-504072df67db" (UID: "d267cc3b-bbd7-42aa-b457-504072df67db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.425187 5399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-tmp-dir\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.425207 5399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-model-cache\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.425223 5399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-home\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.425238 5399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-dshm\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.425251 5399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead97984-d504-497d-b1ca-c2781eace145-home\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.425265 5399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead97984-d504-497d-b1ca-c2781eace145-tls-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.425307 5399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-model-cache\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.425320 5399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-home\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.425332 5399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-tmp-dir\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.425399 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.425347 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpsb2\" (UniqueName: \"kubernetes.io/projected/ead97984-d504-497d-b1ca-c2781eace145-kube-api-access-tpsb2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.427114 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.427086 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d267cc3b-bbd7-42aa-b457-504072df67db-kube-api-access-rrgqx" pod "d267cc3b-bbd7-42aa-b457-504072df67db" (UID: "d267cc3b-bbd7-42aa-b457-504072df67db"). InnerVolumeSpecName "kube-api-access-rrgqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:01.427426 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.427291 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d267cc3b-bbd7-42aa-b457-504072df67db-tls-certs" pod "d267cc3b-bbd7-42aa-b457-504072df67db" (UID: "d267cc3b-bbd7-42aa-b457-504072df67db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:01.427518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.427432 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/765d761d-8567-46a2-a441-c185a232f060-tls-certs" pod "765d761d-8567-46a2-a441-c185a232f060" (UID: "765d761d-8567-46a2-a441-c185a232f060"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:01.427518 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.427441 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-dshm" pod "765d761d-8567-46a2-a441-c185a232f060" (UID: "765d761d-8567-46a2-a441-c185a232f060"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.427623 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.427573 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765d761d-8567-46a2-a441-c185a232f060-kube-api-access-crvlq" pod "765d761d-8567-46a2-a441-c185a232f060" (UID: "765d761d-8567-46a2-a441-c185a232f060"). InnerVolumeSpecName "kube-api-access-crvlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:01.428384 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.428362 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-dshm" pod "d267cc3b-bbd7-42aa-b457-504072df67db" (UID: "d267cc3b-bbd7-42aa-b457-504072df67db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:01.526913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.526881 5399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765d761d-8567-46a2-a441-c185a232f060-dshm\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.526913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.526915 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrgqx\" (UniqueName: \"kubernetes.io/projected/d267cc3b-bbd7-42aa-b457-504072df67db-kube-api-access-rrgqx\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.527064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.526925 5399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765d761d-8567-46a2-a441-c185a232f060-tls-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.527064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.526934 5399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d267cc3b-bbd7-42aa-b457-504072df67db-dshm\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.527064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.526943 5399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d267cc3b-bbd7-42aa-b457-504072df67db-tls-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.527064 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.526951 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crvlq\" (UniqueName: \"kubernetes.io/projected/765d761d-8567-46a2-a441-c185a232f060-kube-api-access-crvlq\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:01.794353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.794249 5399 generic.go:356] "Generic (PLEG): container finished" podID="ead97984-d504-497d-b1ca-c2781eace145" containerID="8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a" exitCode=0 Nov 02 20:44:01.794353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.794316 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" Nov 02 20:44:01.794353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.794333 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" event={"ID":"ead97984-d504-497d-b1ca-c2781eace145","Type":"ContainerDied","Data":"8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a"} Nov 02 20:44:01.794636 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.794374 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4" event={"ID":"ead97984-d504-497d-b1ca-c2781eace145","Type":"ContainerDied","Data":"f8b392b8d81957069be6bffcf42803d3338bc4b766388e4f3174c660fe88824a"} Nov 02 20:44:01.794636 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.794395 5399 scope.go:122] "RemoveContainer" containerID="8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a" Nov 02 20:44:01.795938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.795914 5399 generic.go:356] "Generic (PLEG): container finished" podID="765d761d-8567-46a2-a441-c185a232f060" containerID="5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f" exitCode=0 Nov 02 20:44:01.796083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.795948 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" Nov 02 20:44:01.796083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.795987 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" event={"ID":"765d761d-8567-46a2-a441-c185a232f060","Type":"ContainerDied","Data":"5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f"} Nov 02 20:44:01.796083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.796041 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk" event={"ID":"765d761d-8567-46a2-a441-c185a232f060","Type":"ContainerDied","Data":"2131eda9c5a0785cc6214756eff3e03c5456b00d64811e586786590f163e9b67"} Nov 02 20:44:01.797799 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.797778 5399 generic.go:356] "Generic (PLEG): container finished" podID="d267cc3b-bbd7-42aa-b457-504072df67db" containerID="973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55" exitCode=0 Nov 02 20:44:01.797887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.797812 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" event={"ID":"d267cc3b-bbd7-42aa-b457-504072df67db","Type":"ContainerDied","Data":"973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55"} Nov 02 20:44:01.797887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.797818 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" Nov 02 20:44:01.797887 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.797840 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk" event={"ID":"d267cc3b-bbd7-42aa-b457-504072df67db","Type":"ContainerDied","Data":"72d4b75faf5766e71567ef72e18347d26230f3e9e383523a04137c815a24b5dc"} Nov 02 20:44:01.805497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.805475 5399 scope.go:122] "RemoveContainer" containerID="8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a" Nov 02 20:44:01.805766 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:44:01.805749 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a\": container with ID starting with 8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a not found: ID does not exist" containerID="8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a" Nov 02 20:44:01.805825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.805775 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a"} err="failed to get container status \"8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a\": rpc error: code = NotFound desc = could not find container \"8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a\": container with ID starting with 8b125645613872e94f6a768ea84c98b0f0f0f8d8014ae81fffb40213a0eb539a not found: ID does not exist" Nov 02 20:44:01.805825 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.805792 5399 scope.go:122] "RemoveContainer" containerID="5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f" Nov 02 20:44:01.814876 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.814855 5399 scope.go:122] "RemoveContainer" containerID="5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f" Nov 02 20:44:01.815171 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:44:01.815150 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f\": container with ID starting with 5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f not found: ID does not exist" containerID="5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f" Nov 02 20:44:01.815229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.815181 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f"} err="failed to get container status \"5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f\": rpc error: code = NotFound desc = could not find container \"5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f\": container with ID starting with 5f5095ecc32e5cec8f15e1e19fff9dcd5849ee1061f4576df0ec9a7ffe1a563f not found: ID does not exist" Nov 02 20:44:01.815229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.815199 5399 scope.go:122] "RemoveContainer" containerID="973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55" Nov 02 20:44:01.824080 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.824056 5399 scope.go:122] "RemoveContainer" containerID="973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55" Nov 02 20:44:01.824396 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:44:01.824373 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55\": container with ID starting with 973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55 not found: ID does not exist" containerID="973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55" Nov 02 20:44:01.824467 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.824404 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55"} err="failed to get container status \"973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55\": rpc error: code = NotFound desc = could not find container \"973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55\": container with ID starting with 973e2ec50459a05871e04b50087e1a2bbc1d7c08226eca794515104ff5c12f55 not found: ID does not exist" Nov 02 20:44:01.948779 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.948741 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4"] Nov 02 20:44:01.951204 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:01.951178 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/distinct-2-w0-035e47-kserve-7d46bf98c6-z4qc4"] Nov 02 20:44:02.052416 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:02.052319 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk"] Nov 02 20:44:02.055602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:02.055572 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/simulator-w0-035e47-kserve-7fc4cffb45-pqkpk"] Nov 02 20:44:02.156727 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:02.156691 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk"] Nov 02 20:44:02.161428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:02.161398 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-035e47/premium-w0-035e47-kserve-9f46d69fb-c2fvk"] Nov 02 20:44:03.506433 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:03.506401 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765d761d-8567-46a2-a441-c185a232f060" path="/var/lib/kubelet/pods/765d761d-8567-46a2-a441-c185a232f060/volumes" Nov 02 20:44:03.506846 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:03.506832 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" path="/var/lib/kubelet/pods/d267cc3b-bbd7-42aa-b457-504072df67db/volumes" Nov 02 20:44:03.507199 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:03.507188 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead97984-d504-497d-b1ca-c2781eace145" path="/var/lib/kubelet/pods/ead97984-d504-497d-b1ca-c2781eace145/volumes" Nov 02 20:44:05.920877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:05.920842 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh"] Nov 02 20:44:05.921647 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:05.921613 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" containerID="cri-o://2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f" gracePeriod=30 Nov 02 20:44:07.134564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:07.134526 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:44:07.148294 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:07.148246 5399 patch_prober.go:29] interesting pod/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:44:07.148446 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:07.148323 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:44:11.177418 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.177393 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:44:11.322170 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322139 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-podinfo\") pod \"d5d56f67-d429-4c97-9a92-1c8570de49f8\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " Nov 02 20:44:11.322385 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322211 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-token\" (UniqueName: \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-token\") pod \"d5d56f67-d429-4c97-9a92-1c8570de49f8\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " Nov 02 20:44:11.322385 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322301 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-envoy\") pod \"d5d56f67-d429-4c97-9a92-1c8570de49f8\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " Nov 02 20:44:11.322385 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322348 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d56f67-d429-4c97-9a92-1c8570de49f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d56f67-d429-4c97-9a92-1c8570de49f8-istiod-ca-cert\") pod \"d5d56f67-d429-4c97-9a92-1c8570de49f8\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " Nov 02 20:44:11.322528 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322406 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-certs\") pod \"d5d56f67-d429-4c97-9a92-1c8570de49f8\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " Nov 02 20:44:11.322590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322549 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-socket\") pod \"d5d56f67-d429-4c97-9a92-1c8570de49f8\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " Nov 02 20:44:11.322655 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322601 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-kube-api-access-x8bzd\" (UniqueName: \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-kube-api-access-x8bzd\") pod \"d5d56f67-d429-4c97-9a92-1c8570de49f8\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " Nov 02 20:44:11.322655 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322639 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-credential-socket\") pod \"d5d56f67-d429-4c97-9a92-1c8570de49f8\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " Nov 02 20:44:11.322759 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322676 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-data\") pod \"d5d56f67-d429-4c97-9a92-1c8570de49f8\" (UID: \"d5d56f67-d429-4c97-9a92-1c8570de49f8\") " Nov 02 20:44:11.322759 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322700 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-certs" pod "d5d56f67-d429-4c97-9a92-1c8570de49f8" (UID: "d5d56f67-d429-4c97-9a92-1c8570de49f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:11.322864 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322762 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d56f67-d429-4c97-9a92-1c8570de49f8-istiod-ca-cert" pod "d5d56f67-d429-4c97-9a92-1c8570de49f8" (UID: "d5d56f67-d429-4c97-9a92-1c8570de49f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:11.322864 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.322804 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-socket" pod "d5d56f67-d429-4c97-9a92-1c8570de49f8" (UID: "d5d56f67-d429-4c97-9a92-1c8570de49f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:11.323027 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.323002 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-data" pod "d5d56f67-d429-4c97-9a92-1c8570de49f8" (UID: "d5d56f67-d429-4c97-9a92-1c8570de49f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:11.323160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.323111 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d56f67-d429-4c97-9a92-1c8570de49f8-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:11.323160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.323130 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:11.323160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.323144 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:11.323160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.323157 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:11.323385 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.323300 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-credential-socket" pod "d5d56f67-d429-4c97-9a92-1c8570de49f8" (UID: "d5d56f67-d429-4c97-9a92-1c8570de49f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:11.324606 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.324583 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-podinfo" pod "d5d56f67-d429-4c97-9a92-1c8570de49f8" (UID: "d5d56f67-d429-4c97-9a92-1c8570de49f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:44:11.324803 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.324783 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-kube-api-access-x8bzd" pod "d5d56f67-d429-4c97-9a92-1c8570de49f8" (UID: "d5d56f67-d429-4c97-9a92-1c8570de49f8"). InnerVolumeSpecName "kube-api-access-x8bzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:11.324899 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.324882 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-envoy" pod "d5d56f67-d429-4c97-9a92-1c8570de49f8" (UID: "d5d56f67-d429-4c97-9a92-1c8570de49f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:11.324980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.324963 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-token" pod "d5d56f67-d429-4c97-9a92-1c8570de49f8" (UID: "d5d56f67-d429-4c97-9a92-1c8570de49f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:11.424698 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.424663 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8bzd\" (UniqueName: \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-kube-api-access-x8bzd\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:11.424698 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.424697 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:11.424939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.424711 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:11.424939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.424724 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:11.424939 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.424736 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d56f67-d429-4c97-9a92-1c8570de49f8-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:11.868453 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.868412 5399 generic.go:356] "Generic (PLEG): container finished" podID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerID="2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f" exitCode=0 Nov 02 20:44:11.868645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.868461 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" Nov 02 20:44:11.868645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.868489 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" event={"ID":"d5d56f67-d429-4c97-9a92-1c8570de49f8","Type":"ContainerDied","Data":"2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f"} Nov 02 20:44:11.868645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.868549 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh" event={"ID":"d5d56f67-d429-4c97-9a92-1c8570de49f8","Type":"ContainerDied","Data":"ef4c6d4990f58410761aa5ed58f8f092a73be53b218d989278d7d65c9f132648"} Nov 02 20:44:11.868645 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.868583 5399 scope.go:122] "RemoveContainer" containerID="2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f" Nov 02 20:44:11.878436 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.878410 5399 scope.go:122] "RemoveContainer" containerID="57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727" Nov 02 20:44:11.887815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.887795 5399 scope.go:122] "RemoveContainer" containerID="2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f" Nov 02 20:44:11.888122 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:44:11.888087 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f\": container with ID starting with 2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f not found: ID does not exist" containerID="2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f" Nov 02 20:44:11.888202 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.888124 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f"} err="failed to get container status \"2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f\": rpc error: code = NotFound desc = could not find container \"2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f\": container with ID starting with 2819120e46e726b31916212892300387393b44bf9cb5d1113d750c9e902b213f not found: ID does not exist" Nov 02 20:44:11.888202 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.888140 5399 scope.go:122] "RemoveContainer" containerID="57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727" Nov 02 20:44:11.888475 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:44:11.888456 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727\": container with ID starting with 57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727 not found: ID does not exist" containerID="57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727" Nov 02 20:44:11.888551 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:11.888480 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727"} err="failed to get container status \"57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727\": rpc error: code = NotFound desc = could not find container \"57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727\": container with ID starting with 57536d0a0b7f4d00ba6541e7aad4460d6fbddf2e800532be6120a1cdac561727 not found: ID does not exist" Nov 02 20:44:12.021510 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:12.021467 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh"] Nov 02 20:44:12.068464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:12.068427 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc64d3-openshift-default-785cc8d9ff-zblnh"] Nov 02 20:44:13.507462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:13.507429 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" path="/var/lib/kubelet/pods/d5d56f67-d429-4c97-9a92-1c8570de49f8/volumes" Nov 02 20:44:16.643927 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:16.643891 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv"] Nov 02 20:44:16.644493 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:16.644116 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" containerID="cri-o://92f847533c7ce9efa56b3d612da0c7201bbd3bb3fdf9ff17c1a6545c71a08afb" gracePeriod=30 Nov 02 20:44:18.236516 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:44:18.236474 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:44:21.831704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:21.831670 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-zxn5f"] Nov 02 20:44:21.832191 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:21.831926 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" podUID="ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" containerName="manager" containerID="cri-o://55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39" gracePeriod=30 Nov 02 20:44:21.934639 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:21.934602 5399 generic.go:356] "Generic (PLEG): container finished" podID="d27f161f-eb87-4410-a556-400a65eab338" containerID="92f847533c7ce9efa56b3d612da0c7201bbd3bb3fdf9ff17c1a6545c71a08afb" exitCode=0 Nov 02 20:44:21.934804 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:21.934681 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" event={"ID":"d27f161f-eb87-4410-a556-400a65eab338","Type":"ContainerDied","Data":"92f847533c7ce9efa56b3d612da0c7201bbd3bb3fdf9ff17c1a6545c71a08afb"} Nov 02 20:44:22.001953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.001924 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:44:22.031323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031216 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-socket\") pod \"d27f161f-eb87-4410-a556-400a65eab338\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " Nov 02 20:44:22.031323 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031288 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-certs\") pod \"d27f161f-eb87-4410-a556-400a65eab338\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " Nov 02 20:44:22.031579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031326 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-istio-token\" (UniqueName: \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-istio-token\") pod \"d27f161f-eb87-4410-a556-400a65eab338\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " Nov 02 20:44:22.031579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031379 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d27f161f-eb87-4410-a556-400a65eab338-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27f161f-eb87-4410-a556-400a65eab338-istio-podinfo\") pod \"d27f161f-eb87-4410-a556-400a65eab338\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " Nov 02 20:44:22.031579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031449 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-envoy\") pod \"d27f161f-eb87-4410-a556-400a65eab338\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " Nov 02 20:44:22.031579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031495 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-credential-socket\") pod \"d27f161f-eb87-4410-a556-400a65eab338\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " Nov 02 20:44:22.031579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031523 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-kube-api-access-mdgvc\" (UniqueName: \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-kube-api-access-mdgvc\") pod \"d27f161f-eb87-4410-a556-400a65eab338\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " Nov 02 20:44:22.031579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031521 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-socket" pod "d27f161f-eb87-4410-a556-400a65eab338" (UID: "d27f161f-eb87-4410-a556-400a65eab338"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:22.031579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031558 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-data\") pod \"d27f161f-eb87-4410-a556-400a65eab338\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " Nov 02 20:44:22.031986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031596 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27f161f-eb87-4410-a556-400a65eab338-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27f161f-eb87-4410-a556-400a65eab338-istiod-ca-cert\") pod \"d27f161f-eb87-4410-a556-400a65eab338\" (UID: \"d27f161f-eb87-4410-a556-400a65eab338\") " Nov 02 20:44:22.031986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031651 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-certs" pod "d27f161f-eb87-4410-a556-400a65eab338" (UID: "d27f161f-eb87-4410-a556-400a65eab338"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:22.031986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031951 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.031986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.031971 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.032224 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.032189 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-data" pod "d27f161f-eb87-4410-a556-400a65eab338" (UID: "d27f161f-eb87-4410-a556-400a65eab338"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:22.032318 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.032248 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-credential-socket" pod "d27f161f-eb87-4410-a556-400a65eab338" (UID: "d27f161f-eb87-4410-a556-400a65eab338"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:22.032492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.032465 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27f161f-eb87-4410-a556-400a65eab338-istiod-ca-cert" pod "d27f161f-eb87-4410-a556-400a65eab338" (UID: "d27f161f-eb87-4410-a556-400a65eab338"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:22.034082 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.034057 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d27f161f-eb87-4410-a556-400a65eab338-istio-podinfo" pod "d27f161f-eb87-4410-a556-400a65eab338" (UID: "d27f161f-eb87-4410-a556-400a65eab338"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:44:22.034192 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.034128 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-kube-api-access-mdgvc" pod "d27f161f-eb87-4410-a556-400a65eab338" (UID: "d27f161f-eb87-4410-a556-400a65eab338"). InnerVolumeSpecName "kube-api-access-mdgvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:22.034192 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.034146 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-istio-token" pod "d27f161f-eb87-4410-a556-400a65eab338" (UID: "d27f161f-eb87-4410-a556-400a65eab338"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:22.034417 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.034399 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-envoy" pod "d27f161f-eb87-4410-a556-400a65eab338" (UID: "d27f161f-eb87-4410-a556-400a65eab338"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:22.074630 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.074605 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 20:44:22.132633 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.132606 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-kube-api-access-sc9kq\" (UniqueName: \"kubernetes.io/projected/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-kube-api-access-sc9kq\") pod \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " Nov 02 20:44:22.132786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.132687 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-webhook-cert\") pod \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " Nov 02 20:44:22.132786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.132777 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-metrics-certs\") pod \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\" (UID: \"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036\") " Nov 02 20:44:22.133097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.133073 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27f161f-eb87-4410-a556-400a65eab338-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.133097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.133098 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.133322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.133114 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.133322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.133131 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdgvc\" (UniqueName: \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-kube-api-access-mdgvc\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.133322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.133146 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27f161f-eb87-4410-a556-400a65eab338-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.133322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.133163 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27f161f-eb87-4410-a556-400a65eab338-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.133322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.133177 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27f161f-eb87-4410-a556-400a65eab338-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.134818 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.134794 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-kube-api-access-sc9kq" pod "ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" (UID: "ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036"). InnerVolumeSpecName "kube-api-access-sc9kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:22.134942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.134824 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-metrics-certs" pod "ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" (UID: "ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:22.134942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.134857 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-webhook-cert" pod "ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" (UID: "ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:22.234040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.233997 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.234040 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.234040 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc9kq\" (UniqueName: \"kubernetes.io/projected/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-kube-api-access-sc9kq\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.234324 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.234055 5399 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036-webhook-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:22.941999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.941961 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" Nov 02 20:44:22.941999 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.941986 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv" event={"ID":"d27f161f-eb87-4410-a556-400a65eab338","Type":"ContainerDied","Data":"9d505d928587649cffcacff674f161911478d2dfc63eada8c1a36f7320ae5394"} Nov 02 20:44:22.942632 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.942052 5399 scope.go:122] "RemoveContainer" containerID="92f847533c7ce9efa56b3d612da0c7201bbd3bb3fdf9ff17c1a6545c71a08afb" Nov 02 20:44:22.943934 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.943904 5399 generic.go:356] "Generic (PLEG): container finished" podID="ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" containerID="55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39" exitCode=0 Nov 02 20:44:22.944051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.943948 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" Nov 02 20:44:22.944051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.943949 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" event={"ID":"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036","Type":"ContainerDied","Data":"55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39"} Nov 02 20:44:22.944163 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.944079 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-zxn5f" event={"ID":"ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036","Type":"ContainerDied","Data":"24ca67f2ade289c449c9ff5267de8f310d943ab5d7488222eac10ba06e307792"} Nov 02 20:44:22.953882 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.953860 5399 scope.go:122] "RemoveContainer" containerID="55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39" Nov 02 20:44:22.962945 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.962926 5399 scope.go:122] "RemoveContainer" containerID="e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c" Nov 02 20:44:22.971036 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.971008 5399 scope.go:122] "RemoveContainer" containerID="55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39" Nov 02 20:44:22.971299 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:44:22.971264 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39\": container with ID starting with 55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39 not found: ID does not exist" containerID="55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39" Nov 02 20:44:22.971382 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.971312 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39"} err="failed to get container status \"55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39\": rpc error: code = NotFound desc = could not find container \"55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39\": container with ID starting with 55d0b78481a16d5a01bf31539f208bbeb5436eb42cc257494c61ef4012a91a39 not found: ID does not exist" Nov 02 20:44:22.971382 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.971339 5399 scope.go:122] "RemoveContainer" containerID="e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c" Nov 02 20:44:22.971589 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:44:22.971572 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c\": container with ID starting with e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c not found: ID does not exist" containerID="e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c" Nov 02 20:44:22.971630 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:22.971598 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c"} err="failed to get container status \"e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c\": rpc error: code = NotFound desc = could not find container \"e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c\": container with ID starting with e539d0368512392f0ae825ea8811b497602e8a06b9cc2ffa439bd78f3328824c not found: ID does not exist" Nov 02 20:44:23.095083 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.095044 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-zxn5f"] Nov 02 20:44:23.141525 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.141489 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-zxn5f"] Nov 02 20:44:23.199451 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.199412 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv"] Nov 02 20:44:23.202184 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.202157 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-035e47-openshift-default-67d4d9cd57-5qqpv"] Nov 02 20:44:23.514084 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.514001 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27f161f-eb87-4410-a556-400a65eab338" path="/var/lib/kubelet/pods/d27f161f-eb87-4410-a556-400a65eab338/volumes" Nov 02 20:44:23.514426 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.514412 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" path="/var/lib/kubelet/pods/ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036/volumes" Nov 02 20:44:23.941756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.941716 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-94xbs"] Nov 02 20:44:23.942115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942102 5399 cpu_manager.go:414] "containerMap: removing container" podUID="20aa971f-7e5c-4267-a7a8-b07b99676807" containerName="payload-processing" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942116 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20aa971f-7e5c-4267-a7a8-b07b99676807" containerName="payload-processing" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942131 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942136 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942149 5399 cpu_manager.go:414] "containerMap: removing container" podUID="f04cadfc-8aff-42f7-b415-8fe0aa34a767" containerName="revoke-keys" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942154 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04cadfc-8aff-42f7-b415-8fe0aa34a767" containerName="revoke-keys" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942161 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" containerName="manager" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942166 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" containerName="manager" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942173 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942177 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942185 5399 cpu_manager.go:414] "containerMap: removing container" podUID="90935866-4032-47ca-8938-c965d81fdda0" containerName="payload-pre-processing" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942190 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90935866-4032-47ca-8938-c965d81fdda0" containerName="payload-pre-processing" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942207 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942211 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942223 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942227 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942236 5399 cpu_manager.go:414] "containerMap: removing container" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942240 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942246 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942251 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942340 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d267cc3b-bbd7-42aa-b457-504072df67db" containerName="main" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942352 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942358 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" containerName="manager" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942363 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="765d761d-8567-46a2-a441-c185a232f060" containerName="main" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942370 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27f161f-eb87-4410-a556-400a65eab338" containerName="istio-proxy" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942377 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="20aa971f-7e5c-4267-a7a8-b07b99676807" containerName="payload-processing" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942385 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04cadfc-8aff-42f7-b415-8fe0aa34a767" containerName="revoke-keys" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942390 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" containerName="manager" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942396 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead97984-d504-497d-b1ca-c2781eace145" containerName="main" Nov 02 20:44:23.942470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.942401 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="90935866-4032-47ca-8938-c965d81fdda0" containerName="payload-pre-processing" Nov 02 20:44:23.947625 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.947599 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:23.996507 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:23.996477 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 20:44:24.042888 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.042853 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 20:44:24.043066 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.042885 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 20:44:24.043066 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.042990 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h8zjd\"" Nov 02 20:44:24.045504 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.045483 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-94xbs"] Nov 02 20:44:24.052392 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.052358 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wbl\" (UniqueName: \"kubernetes.io/projected/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-kube-api-access-85wbl\") pod \"maas-controller-5ff67f6f5f-94xbs\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:24.052544 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.052422 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-metrics-certs\") pod \"maas-controller-5ff67f6f5f-94xbs\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:24.053259 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.052786 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-webhook-cert\") pod \"maas-controller-5ff67f6f5f-94xbs\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:24.153945 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.153908 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wbl\" (UniqueName: \"kubernetes.io/projected/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-kube-api-access-85wbl\") pod \"maas-controller-5ff67f6f5f-94xbs\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:24.154137 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.153957 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-metrics-certs\") pod \"maas-controller-5ff67f6f5f-94xbs\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:24.154137 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.153983 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-webhook-cert\") pod \"maas-controller-5ff67f6f5f-94xbs\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:24.156582 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.156549 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-webhook-cert\") pod \"maas-controller-5ff67f6f5f-94xbs\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:24.156774 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.156755 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-metrics-certs\") pod \"maas-controller-5ff67f6f5f-94xbs\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:24.207698 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.207626 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wbl\" (UniqueName: \"kubernetes.io/projected/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-kube-api-access-85wbl\") pod \"maas-controller-5ff67f6f5f-94xbs\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:24.261803 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.261765 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:24.632051 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.632016 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-94xbs"] Nov 02 20:44:24.665176 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:44:24.665144 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7322ef78_2683_4f47_a7ef_1c850bbb4d9f.slice/crio-ad903475c494ba61706c6cbfa731835d177104fbdee9ebaf9e0526f6f3444e3f WatchSource:0}: Error finding container ad903475c494ba61706c6cbfa731835d177104fbdee9ebaf9e0526f6f3444e3f: Status 404 returned error can't find the container with id ad903475c494ba61706c6cbfa731835d177104fbdee9ebaf9e0526f6f3444e3f Nov 02 20:44:24.666549 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.666528 5399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:44:24.962395 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:24.962308 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" event={"ID":"7322ef78-2683-4f47-a7ef-1c850bbb4d9f","Type":"ContainerStarted","Data":"ad903475c494ba61706c6cbfa731835d177104fbdee9ebaf9e0526f6f3444e3f"} Nov 02 20:44:25.969977 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:25.969893 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" event={"ID":"7322ef78-2683-4f47-a7ef-1c850bbb4d9f","Type":"ContainerStarted","Data":"2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295"} Nov 02 20:44:25.970388 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:25.970027 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:26.076812 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:26.076757 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" podStartSLOduration=2.15884702 podStartE2EDuration="3.076742883s" totalImagesPullingTime="917.895863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:44:26.075633669 +0000 UTC m=+3108.686285761" watchObservedRunningTime="2026-11-02 20:44:26.076742883 +0000 UTC m=+3108.687394992" Nov 02 20:44:34.044011 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:34.043972 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l"] Nov 02 20:44:34.090997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:34.090963 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc001fc9-pv72l"] Nov 02 20:44:34.356254 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:34.356219 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:44:35.507314 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:35.507257 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4347a86f-c4c4-45cf-b812-4d1e79e4d0bc" path="/var/lib/kubelet/pods/4347a86f-c4c4-45cf-b812-4d1e79e4d0bc/volumes" Nov 02 20:44:36.978511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:36.978480 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:47.043049 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:47.043005 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc"] Nov 02 20:44:47.089377 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:47.089339 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a206ebf6-g2bzc"] Nov 02 20:44:47.506058 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:47.506024 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555b5b76-927c-4248-8a59-5a4cc6f68dcb" path="/var/lib/kubelet/pods/555b5b76-927c-4248-8a59-5a4cc6f68dcb/volumes" Nov 02 20:44:48.236991 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:44:48.236944 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:44:58.056025 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.055988 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-94xbs"] Nov 02 20:44:58.056540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.056185 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" podUID="7322ef78-2683-4f47-a7ef-1c850bbb4d9f" containerName="manager" containerID="cri-o://2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295" gracePeriod=30 Nov 02 20:44:58.308539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.308476 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:58.377474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.377435 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-metrics-certs\") pod \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " Nov 02 20:44:58.377673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.377483 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-kube-api-access-85wbl\" (UniqueName: \"kubernetes.io/projected/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-kube-api-access-85wbl\") pod \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " Nov 02 20:44:58.377673 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.377580 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-webhook-cert\") pod \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\" (UID: \"7322ef78-2683-4f47-a7ef-1c850bbb4d9f\") " Nov 02 20:44:58.379547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.379515 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-metrics-certs" pod "7322ef78-2683-4f47-a7ef-1c850bbb4d9f" (UID: "7322ef78-2683-4f47-a7ef-1c850bbb4d9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:58.379683 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.379608 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-webhook-cert" pod "7322ef78-2683-4f47-a7ef-1c850bbb4d9f" (UID: "7322ef78-2683-4f47-a7ef-1c850bbb4d9f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:58.379746 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.379677 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-kube-api-access-85wbl" pod "7322ef78-2683-4f47-a7ef-1c850bbb4d9f" (UID: "7322ef78-2683-4f47-a7ef-1c850bbb4d9f"). InnerVolumeSpecName "kube-api-access-85wbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:58.479669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.479624 5399 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-webhook-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:58.479669 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.479672 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:58.479892 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:58.479690 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wbl\" (UniqueName: \"kubernetes.io/projected/7322ef78-2683-4f47-a7ef-1c850bbb4d9f-kube-api-access-85wbl\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:59.165556 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:59.165518 5399 generic.go:356] "Generic (PLEG): container finished" podID="7322ef78-2683-4f47-a7ef-1c850bbb4d9f" containerID="2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295" exitCode=0 Nov 02 20:44:59.166039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:59.165565 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" event={"ID":"7322ef78-2683-4f47-a7ef-1c850bbb4d9f","Type":"ContainerDied","Data":"2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295"} Nov 02 20:44:59.166039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:59.165607 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" event={"ID":"7322ef78-2683-4f47-a7ef-1c850bbb4d9f","Type":"ContainerDied","Data":"ad903475c494ba61706c6cbfa731835d177104fbdee9ebaf9e0526f6f3444e3f"} Nov 02 20:44:59.166039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:59.165627 5399 scope.go:122] "RemoveContainer" containerID="2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295" Nov 02 20:44:59.166039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:59.165569 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff67f6f5f-94xbs" Nov 02 20:44:59.175328 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:59.175309 5399 scope.go:122] "RemoveContainer" containerID="2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295" Nov 02 20:44:59.175609 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:44:59.175591 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295\": container with ID starting with 2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295 not found: ID does not exist" containerID="2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295" Nov 02 20:44:59.175674 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:59.175616 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295"} err="failed to get container status \"2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295\": rpc error: code = NotFound desc = could not find container \"2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295\": container with ID starting with 2a8574d5bf24a37d34935f518984a6456de617407ff0652003d2fafd5ad0c295 not found: ID does not exist" Nov 02 20:44:59.318579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:59.318545 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-94xbs"] Nov 02 20:44:59.365154 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:59.365114 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-94xbs"] Nov 02 20:44:59.506187 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:44:59.506102 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7322ef78-2683-4f47-a7ef-1c850bbb4d9f" path="/var/lib/kubelet/pods/7322ef78-2683-4f47-a7ef-1c850bbb4d9f/volumes" Nov 02 20:45:00.837539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.837502 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-vrkd6"] Nov 02 20:45:00.837911 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.837877 5399 cpu_manager.go:414] "containerMap: removing container" podUID="7322ef78-2683-4f47-a7ef-1c850bbb4d9f" containerName="manager" Nov 02 20:45:00.837911 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.837887 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7322ef78-2683-4f47-a7ef-1c850bbb4d9f" containerName="manager" Nov 02 20:45:00.837997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.837914 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" containerName="manager" Nov 02 20:45:00.837997 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.837920 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbf52da-05a4-42fb-b6c2-c3ebc3ec2036" containerName="manager" Nov 02 20:45:00.838071 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.837999 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d56f67-d429-4c97-9a92-1c8570de49f8" containerName="istio-proxy" Nov 02 20:45:00.838071 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.838009 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7322ef78-2683-4f47-a7ef-1c850bbb4d9f" containerName="manager" Nov 02 20:45:00.842478 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.842460 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:00.900746 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.900715 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f14ae96b-1f8a-4b79-9984-d6340f921da1-metrics-certs\") pod \"maas-controller-5ff67f6f5f-vrkd6\" (UID: \"f14ae96b-1f8a-4b79-9984-d6340f921da1\") " pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:00.900915 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.900846 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96rtj\" (UniqueName: \"kubernetes.io/projected/f14ae96b-1f8a-4b79-9984-d6340f921da1-kube-api-access-96rtj\") pod \"maas-controller-5ff67f6f5f-vrkd6\" (UID: \"f14ae96b-1f8a-4b79-9984-d6340f921da1\") " pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:00.900973 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.900914 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f14ae96b-1f8a-4b79-9984-d6340f921da1-webhook-cert\") pod \"maas-controller-5ff67f6f5f-vrkd6\" (UID: \"f14ae96b-1f8a-4b79-9984-d6340f921da1\") " pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:00.938788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.938747 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 20:45:00.938788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.938777 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h8zjd\"" Nov 02 20:45:00.938788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.938784 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 20:45:00.939019 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.938831 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 20:45:00.941046 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:00.941028 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-vrkd6"] Nov 02 20:45:01.001753 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:01.001720 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f14ae96b-1f8a-4b79-9984-d6340f921da1-metrics-certs\") pod \"maas-controller-5ff67f6f5f-vrkd6\" (UID: \"f14ae96b-1f8a-4b79-9984-d6340f921da1\") " pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:01.001931 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:01.001802 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96rtj\" (UniqueName: \"kubernetes.io/projected/f14ae96b-1f8a-4b79-9984-d6340f921da1-kube-api-access-96rtj\") pod \"maas-controller-5ff67f6f5f-vrkd6\" (UID: \"f14ae96b-1f8a-4b79-9984-d6340f921da1\") " pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:01.001931 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:01.001846 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f14ae96b-1f8a-4b79-9984-d6340f921da1-webhook-cert\") pod \"maas-controller-5ff67f6f5f-vrkd6\" (UID: \"f14ae96b-1f8a-4b79-9984-d6340f921da1\") " pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:01.004193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:01.004165 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f14ae96b-1f8a-4b79-9984-d6340f921da1-webhook-cert\") pod \"maas-controller-5ff67f6f5f-vrkd6\" (UID: \"f14ae96b-1f8a-4b79-9984-d6340f921da1\") " pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:01.004330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:01.004216 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f14ae96b-1f8a-4b79-9984-d6340f921da1-metrics-certs\") pod \"maas-controller-5ff67f6f5f-vrkd6\" (UID: \"f14ae96b-1f8a-4b79-9984-d6340f921da1\") " pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:01.055577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:01.055539 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96rtj\" (UniqueName: \"kubernetes.io/projected/f14ae96b-1f8a-4b79-9984-d6340f921da1-kube-api-access-96rtj\") pod \"maas-controller-5ff67f6f5f-vrkd6\" (UID: \"f14ae96b-1f8a-4b79-9984-d6340f921da1\") " pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:01.155250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:01.155156 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:01.519530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:01.519446 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff67f6f5f-vrkd6"] Nov 02 20:45:01.552333 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:45:01.552298 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14ae96b_1f8a_4b79_9984_d6340f921da1.slice/crio-00d112901997e1c31d11430ced3780d077663d4959ca6a60ccaa73c1a52c29e6 WatchSource:0}: Error finding container 00d112901997e1c31d11430ced3780d077663d4959ca6a60ccaa73c1a52c29e6: Status 404 returned error can't find the container with id 00d112901997e1c31d11430ced3780d077663d4959ca6a60ccaa73c1a52c29e6 Nov 02 20:45:02.185809 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:02.185768 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" event={"ID":"f14ae96b-1f8a-4b79-9984-d6340f921da1","Type":"ContainerStarted","Data":"00d112901997e1c31d11430ced3780d077663d4959ca6a60ccaa73c1a52c29e6"} Nov 02 20:45:03.193721 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:03.193677 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" event={"ID":"f14ae96b-1f8a-4b79-9984-d6340f921da1","Type":"ContainerStarted","Data":"db70bdf7b8613022b0fbf447c607d566a2268eaccf56b2d2636d4ce573e8cf7f"} Nov 02 20:45:03.194250 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:03.193834 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:03.304410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:03.304347 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" podStartSLOduration=2.417270629 podStartE2EDuration="3.304332079s" totalImagesPullingTime="887.06145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:03.301190093 +0000 UTC m=+3145.911842175" watchObservedRunningTime="2026-11-02 20:45:03.304332079 +0000 UTC m=+3145.914984169" Nov 02 20:45:14.201742 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:14.201713 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5ff67f6f5f-vrkd6" Nov 02 20:45:18.235966 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:45:18.235880 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:45:39.419183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:39.419143 5399 scope.go:122] "RemoveContainer" containerID="28784316d65c91b938963531ddf55d3afb47cbd1cfbdab8a2186e8627b5e0843" Nov 02 20:45:46.333115 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:45:46.333077 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:45:48.242324 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:45:48.242254 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:46:08.137698 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:08.137659 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:46:15.241208 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.241169 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs"] Nov 02 20:46:15.244968 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.244946 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.343334 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.343297 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs"] Nov 02 20:46:15.405503 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.405466 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.405689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.405513 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.405689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.405540 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc88c07-4a0d-4472-bf37-b35b17b58759-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.405689 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.405653 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.405808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.405742 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kllkx\" (UniqueName: \"kubernetes.io/projected/2fc88c07-4a0d-4472-bf37-b35b17b58759-kube-api-access-kllkx\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.405808 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.405772 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.405875 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.405809 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.405875 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.405868 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.405943 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.405890 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.506908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.506832 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kllkx\" (UniqueName: \"kubernetes.io/projected/2fc88c07-4a0d-4472-bf37-b35b17b58759-kube-api-access-kllkx\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.506908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.506868 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.506908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.506893 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.506920 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.506936 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.506959 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.506983 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.507006 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc88c07-4a0d-4472-bf37-b35b17b58759-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507198 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.507104 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.507265 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.507464 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.507492 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507682 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.507660 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.507845 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.507828 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fc88c07-4a0d-4472-bf37-b35b17b58759-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.509344 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.509322 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.509523 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.509504 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.559839 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.559807 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fc88c07-4a0d-4472-bf37-b35b17b58759-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.560011 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.559900 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kllkx\" (UniqueName: \"kubernetes.io/projected/2fc88c07-4a0d-4472-bf37-b35b17b58759-kube-api-access-kllkx\") pod \"maas-default-gateway-openshift-default-696f47c6c-zfbjs\" (UID: \"2fc88c07-4a0d-4472-bf37-b35b17b58759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:15.857912 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:15.857873 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:16.271716 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:16.271628 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs"] Nov 02 20:46:16.308218 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:46:16.308182 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc88c07_4a0d_4472_bf37_b35b17b58759.slice/crio-42403a73183c54c6066f0e161d4fda97eaec8de05aa40bfede91576502d3ab81 WatchSource:0}: Error finding container 42403a73183c54c6066f0e161d4fda97eaec8de05aa40bfede91576502d3ab81: Status 404 returned error can't find the container with id 42403a73183c54c6066f0e161d4fda97eaec8de05aa40bfede91576502d3ab81 Nov 02 20:46:16.310413 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:16.310380 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:46:16.310505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:16.310449 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:46:16.310505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:16.310480 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:46:16.618742 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:16.618702 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" event={"ID":"2fc88c07-4a0d-4472-bf37-b35b17b58759","Type":"ContainerStarted","Data":"0ba1c54f2be142c27d9e533deb473c127343e7f5fdb531eb329b240c6337d005"} Nov 02 20:46:16.618927 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:16.618759 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" event={"ID":"2fc88c07-4a0d-4472-bf37-b35b17b58759","Type":"ContainerStarted","Data":"42403a73183c54c6066f0e161d4fda97eaec8de05aa40bfede91576502d3ab81"} Nov 02 20:46:16.728711 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:16.728656 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" podStartSLOduration=1.728641213 podStartE2EDuration="1.728641213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:16.726345397 +0000 UTC m=+3219.336997502" watchObservedRunningTime="2026-11-02 20:46:16.728641213 +0000 UTC m=+3219.339293300" Nov 02 20:46:16.858831 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:16.858778 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:17.859940 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:17.859906 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zfbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 20:46:17.860474 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:17.859959 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" podUID="2fc88c07-4a0d-4472-bf37-b35b17b58759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 20:46:18.261931 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:46:18.261851 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:46:18.859177 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:18.859142 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zfbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 20:46:18.859398 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:18.859193 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" podUID="2fc88c07-4a0d-4472-bf37-b35b17b58759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 20:46:18.986206 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:18.986166 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:46:19.858550 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:19.858508 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zfbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 20:46:19.858791 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:19.858571 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" podUID="2fc88c07-4a0d-4472-bf37-b35b17b58759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 20:46:20.858380 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:20.858336 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zfbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 20:46:20.858804 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:20.858397 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" podUID="2fc88c07-4a0d-4472-bf37-b35b17b58759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 20:46:21.858487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:21.858449 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zfbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 20:46:21.858897 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:21.858532 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" podUID="2fc88c07-4a0d-4472-bf37-b35b17b58759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 20:46:22.859015 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:22.858975 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-zfbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Nov 02 20:46:22.859455 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:22.859058 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" podUID="2fc88c07-4a0d-4472-bf37-b35b17b58759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Nov 02 20:46:24.555088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:24.555048 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:24.555622 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:24.555414 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:24.556220 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:24.556201 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-zfbjs" Nov 02 20:46:39.044587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:39.044551 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42"] Nov 02 20:46:39.091190 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:39.091152 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6ab7818-qgv42"] Nov 02 20:46:39.445270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:39.445237 5399 scope.go:122] "RemoveContainer" containerID="25b6b192c71130a2b067698f7d305e1ee9a1e14ac533a7e12a7df0e624f07213" Nov 02 20:46:39.454530 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:39.454507 5399 scope.go:122] "RemoveContainer" containerID="a1520abf5427da0839c5354d2389c2e454c0c2189823545054b238a60e197ca8" Nov 02 20:46:39.506505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:39.506474 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b77ce9-1949-4d3c-a578-c0e44d631590" path="/var/lib/kubelet/pods/80b77ce9-1949-4d3c-a578-c0e44d631590/volumes" Nov 02 20:46:41.520938 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:41.520904 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:46:45.043100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:45.043058 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s"] Nov 02 20:46:45.089547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:45.089510 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddc3fd7d-8t76s"] Nov 02 20:46:45.506155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:45.506120 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b64cbd6-0cd3-487a-b1ff-15b0b6392865" path="/var/lib/kubelet/pods/9b64cbd6-0cd3-487a-b1ff-15b0b6392865/volumes" Nov 02 20:46:48.235554 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:46:48.235468 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:46:52.043005 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:52.042969 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s"] Nov 02 20:46:52.089176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:52.089143 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7774ff05-wqz2s"] Nov 02 20:46:53.506075 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:53.506032 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d65f651-b3e0-4ae1-be4e-8e568705dfdd" path="/var/lib/kubelet/pods/2d65f651-b3e0-4ae1-be4e-8e568705dfdd/volumes" Nov 02 20:46:58.240590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:46:58.240554 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:47:00.620729 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.620690 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj"] Nov 02 20:47:00.626124 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.626100 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.638479 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.638448 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj"] Nov 02 20:47:00.779555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.779516 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c28a1f27-d0ed-412c-8dbb-17429467209a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.779555 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.779561 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.779856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.779587 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.779856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.779697 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.779856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.779718 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.779856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.779743 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.779856 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.779839 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.780162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.779870 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcnfm\" (UniqueName: \"kubernetes.io/projected/c28a1f27-d0ed-412c-8dbb-17429467209a-kube-api-access-bcnfm\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.780162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.779928 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.880942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.880824 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.880942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.880894 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c28a1f27-d0ed-412c-8dbb-17429467209a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.880942 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.880931 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.881256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.880960 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.881256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.881015 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.881256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.881041 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.881256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.881082 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.881256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.881163 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.881256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.881194 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcnfm\" (UniqueName: \"kubernetes.io/projected/c28a1f27-d0ed-412c-8dbb-17429467209a-kube-api-access-bcnfm\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.881614 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.881372 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.881683 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.881650 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.882128 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.881905 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.882547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.882521 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.882681 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.882661 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c28a1f27-d0ed-412c-8dbb-17429467209a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.884830 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.884790 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.885039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.885021 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.934576 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.934539 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c28a1f27-d0ed-412c-8dbb-17429467209a-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.934778 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.934557 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcnfm\" (UniqueName: \"kubernetes.io/projected/c28a1f27-d0ed-412c-8dbb-17429467209a-kube-api-access-bcnfm\") pod \"maas-default-gateway-openshift-default-696f47c6c-f6jgj\" (UID: \"c28a1f27-d0ed-412c-8dbb-17429467209a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:00.955592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:00.955548 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:01.379986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:01.379960 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj"] Nov 02 20:47:01.381587 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:47:01.381559 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28a1f27_d0ed_412c_8dbb_17429467209a.slice/crio-38ddcda2fb38096f69cfaf0ccbcbb6bb8e6eb43a741f6490d286b87dfe28c537 WatchSource:0}: Error finding container 38ddcda2fb38096f69cfaf0ccbcbb6bb8e6eb43a741f6490d286b87dfe28c537: Status 404 returned error can't find the container with id 38ddcda2fb38096f69cfaf0ccbcbb6bb8e6eb43a741f6490d286b87dfe28c537 Nov 02 20:47:01.383878 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:01.383840 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:47:01.383972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:01.383926 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:47:01.384024 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:01.383968 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:47:01.904092 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:01.904060 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" event={"ID":"c28a1f27-d0ed-412c-8dbb-17429467209a","Type":"ContainerStarted","Data":"7e22e10e954ddf5de08911e31e307b63dd045ca4d7e71d0ba1ed399316e5e4fd"} Nov 02 20:47:01.904511 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:01.904115 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" event={"ID":"c28a1f27-d0ed-412c-8dbb-17429467209a","Type":"ContainerStarted","Data":"38ddcda2fb38096f69cfaf0ccbcbb6bb8e6eb43a741f6490d286b87dfe28c537"} Nov 02 20:47:01.956375 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:01.956337 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:02.105422 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:02.105358 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" podStartSLOduration=2.10534205 podStartE2EDuration="2.10534205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:02.058247685 +0000 UTC m=+3264.668899788" watchObservedRunningTime="2026-11-02 20:47:02.10534205 +0000 UTC m=+3264.715994141" Nov 02 20:47:02.958394 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:02.958362 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-f6jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 02 20:47:02.958892 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:02.958416 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" podUID="c28a1f27-d0ed-412c-8dbb-17429467209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 02 20:47:03.956908 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:03.956867 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-f6jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 02 20:47:03.957106 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:03.956927 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" podUID="c28a1f27-d0ed-412c-8dbb-17429467209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 02 20:47:04.956038 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:04.955999 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-f6jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 02 20:47:04.956481 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:04.956059 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" podUID="c28a1f27-d0ed-412c-8dbb-17429467209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 02 20:47:05.052368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:05.052317 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77"] Nov 02 20:47:05.057826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:05.057800 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644b8b58-dck77"] Nov 02 20:47:05.506642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:05.506610 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356cea95-7878-4474-b71c-75d02dfc9e00" path="/var/lib/kubelet/pods/356cea95-7878-4474-b71c-75d02dfc9e00/volumes" Nov 02 20:47:05.956943 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:05.956905 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-f6jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 02 20:47:05.957381 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:05.956964 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" podUID="c28a1f27-d0ed-412c-8dbb-17429467209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 02 20:47:06.956233 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:06.956193 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-f6jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 02 20:47:06.956497 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:06.956244 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" podUID="c28a1f27-d0ed-412c-8dbb-17429467209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 02 20:47:07.956491 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:07.956452 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-f6jgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Nov 02 20:47:07.958960 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:07.956503 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" podUID="c28a1f27-d0ed-412c-8dbb-17429467209a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Nov 02 20:47:09.767120 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:09.767090 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:09.767653 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:09.767575 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:09.768351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:09.768323 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-f6jgj" Nov 02 20:47:10.053193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:10.053105 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf"] Nov 02 20:47:10.057256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:10.057222 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2"] Nov 02 20:47:10.060891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:10.060863 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cab50db4-jrsgf"] Nov 02 20:47:10.075372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:10.075343 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53183fcb-xlch2"] Nov 02 20:47:11.505980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:11.505946 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246aea8c-620e-49ca-9507-9e5ce27202f5" path="/var/lib/kubelet/pods/246aea8c-620e-49ca-9507-9e5ce27202f5/volumes" Nov 02 20:47:11.506435 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:11.506423 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d29d801d-a2f2-4886-b9a6-06abba1407f7" path="/var/lib/kubelet/pods/d29d801d-a2f2-4886-b9a6-06abba1407f7/volumes" Nov 02 20:47:18.236560 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:47:18.236522 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:47:38.828679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:38.828644 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:47:39.480677 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:39.480638 5399 scope.go:122] "RemoveContainer" containerID="7dfbdd1f653c19b277be1182f5f4ad3cd9898f7c6b468239a9126e315e26b43a" Nov 02 20:47:39.491879 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:39.491858 5399 scope.go:122] "RemoveContainer" containerID="cba402d0fbc543c53245d74ef16f8e49c0a5f53db22e9475399f117e049d955c" Nov 02 20:47:48.236346 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:47:48.236311 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:47:49.751913 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:49.751875 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:47:57.698460 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:57.698425 5399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-g7smq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:47:57.698923 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:47:57.698567 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-g7smq" podUID="9953c331-dda5-4f9e-b084-27e3b2caa37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:48:01.363829 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:01.363789 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:48:18.236507 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:48:18.236417 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:48:24.341146 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:24.341109 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:48:34.153786 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:34.153746 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:48:39.525367 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:39.525335 5399 scope.go:122] "RemoveContainer" containerID="99191506fb81b80d1e51f98cfaced5824b5dfe18bf6c6ff07578fc0a1d6cc689" Nov 02 20:48:39.537219 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:39.537192 5399 scope.go:122] "RemoveContainer" containerID="48227cc9ead8aa035a64ed4a6d08fdc2f689fa47bf273a0d00da8b6c5da58c35" Nov 02 20:48:39.545804 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:39.545782 5399 scope.go:122] "RemoveContainer" containerID="527a3deb500556eb77c43aa25b7913c262ff093f311feeab96fa56d97ae77ea4" Nov 02 20:48:39.553994 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:39.553976 5399 scope.go:122] "RemoveContainer" containerID="a5c39c4f2c8cbae4d48678dfe5fd799a9ed3cd2ad7f2e628d92c7a014fa075df" Nov 02 20:48:39.562652 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:39.562634 5399 scope.go:122] "RemoveContainer" containerID="88fc518f6d98c6ed09129935fcd13dc4315c6d589562de4fe93d4ce78d61c87a" Nov 02 20:48:39.570243 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:39.570223 5399 scope.go:122] "RemoveContainer" containerID="f3f8d92f910494b64544836beedbc96a22af881f3185df91a260e09bb834ca2b" Nov 02 20:48:39.577668 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:39.577640 5399 scope.go:122] "RemoveContainer" containerID="9f50efa74372afe2f1466406651e619ebd967a42491d413e3e76b82e4931157b" Nov 02 20:48:42.042703 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:42.042659 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss"] Nov 02 20:48:42.088896 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:42.088859 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e87257-dxhss"] Nov 02 20:48:43.507543 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:43.507505 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2421f9a-57bf-49e9-90b5-6ac6b35a503b" path="/var/lib/kubelet/pods/a2421f9a-57bf-49e9-90b5-6ac6b35a503b/volumes" Nov 02 20:48:48.236137 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:48:48.236101 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:48:52.157699 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:48:52.157662 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:49:01.047050 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:49:01.047006 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm"] Nov 02 20:49:01.097032 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:49:01.096987 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ff4d7d0-br7nm"] Nov 02 20:49:01.508698 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:49:01.508661 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04cadfc-8aff-42f7-b415-8fe0aa34a767" path="/var/lib/kubelet/pods/f04cadfc-8aff-42f7-b415-8fe0aa34a767/volumes" Nov 02 20:49:05.132133 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:49:05.132096 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:49:18.237574 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:49:18.237527 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:49:27.643925 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:49:27.643894 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:49:39.621358 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:49:39.621323 5399 scope.go:122] "RemoveContainer" containerID="a0ac1e81dd04fe4934ed023a3df95e54012ec013d7707d1815589e891ff0d79d" Nov 02 20:49:39.630268 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:49:39.630248 5399 scope.go:122] "RemoveContainer" containerID="9ca0148cde2100f47281a3ff3d4c66ac42bad537946aaa44d3af89eab8a75da2" Nov 02 20:49:39.638387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:49:39.638370 5399 scope.go:122] "RemoveContainer" containerID="cdf1a73c58aa32667ff1d85e089804440a2637854ad4888aa6c53b756d5c3a00" Nov 02 20:49:39.646176 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:49:39.646156 5399 scope.go:122] "RemoveContainer" containerID="08ce85532becbc6a8623e0564c2efc30df91786c123abc0b8302a2b1cb0a5e57" Nov 02 20:49:48.236160 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:49:48.236073 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:49:49.437121 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:49:49.437077 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:50:12.341088 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:50:12.341051 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:50:18.237377 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:50:18.237339 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:50:23.665192 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:50:23.665158 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:50:39.677776 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:50:39.677743 5399 scope.go:122] "RemoveContainer" containerID="b2dc56410f129134745859976f26ec4d98b5ff111d241b0905b8147aebd07eae" Nov 02 20:50:44.370520 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:50:44.370477 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:50:48.235856 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:50:48.235822 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:50:54.850338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:50:54.850294 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:51:18.237398 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:51:18.237303 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:51:21.131955 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:51:21.131915 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:51:48.236199 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:51:48.236155 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:51:52.542430 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:51:52.542388 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:52:03.323509 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:03.323471 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:52:18.237072 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:52:18.237028 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:52:31.831667 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:31.831626 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:52:42.522231 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:42.522194 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:52:48.236310 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:52:48.236202 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:52:49.521740 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.521708 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw"] Nov 02 20:52:49.524933 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.524913 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.552182 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.552152 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-data\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.552306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.552195 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-certs\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.552306 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.552224 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-credential-socket\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.552393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.552342 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istiod-ca-cert\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.552393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.552378 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-podinfo\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.552492 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.552470 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-envoy\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.552539 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.552529 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-token\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.552579 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.552551 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-socket\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.552619 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.552580 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl9gg\" (UniqueName: \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-kube-api-access-zl9gg\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.586806 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.586779 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-0a02f7-openshift-default-dockercfg-vlcrk\"" Nov 02 20:52:49.622720 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.622688 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw"] Nov 02 20:52:49.653591 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.653561 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-envoy\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.653768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.653613 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-token\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.653768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.653636 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-socket\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.653768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.653660 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl9gg\" (UniqueName: \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-kube-api-access-zl9gg\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.653768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.653686 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-data\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.653768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.653712 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-certs\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.654067 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.653853 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-credential-socket\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.654067 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.653933 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istiod-ca-cert\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.654067 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.653979 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-podinfo\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.654255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.654084 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-socket\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.654255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.654132 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-data\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.654255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.654171 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-certs\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.654255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.654230 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-credential-socket\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.654793 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.654768 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istiod-ca-cert\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.656097 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.656074 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-envoy\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.656462 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.656442 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-podinfo\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.707054 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.707019 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl9gg\" (UniqueName: \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-kube-api-access-zl9gg\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.707226 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.707109 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-token\") pod \"e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:49.838286 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:49.838234 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:50.254421 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:50.254393 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw"] Nov 02 20:52:50.255842 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:52:50.255815 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode436bf1e_d9d5_4bb1_a4e1_f1cf5a607bbe.slice/crio-abbf31db920a52c9f330dda4099afadcc293e363140e85acddf3c78f6e6bbd94 WatchSource:0}: Error finding container abbf31db920a52c9f330dda4099afadcc293e363140e85acddf3c78f6e6bbd94: Status 404 returned error can't find the container with id abbf31db920a52c9f330dda4099afadcc293e363140e85acddf3c78f6e6bbd94 Nov 02 20:52:50.257953 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:50.257922 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:52:50.258013 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:50.257985 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:52:50.258057 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:50.258013 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:52:50.990806 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:50.990766 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" event={"ID":"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe","Type":"ContainerStarted","Data":"be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a"} Nov 02 20:52:50.991214 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:50.990825 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" event={"ID":"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe","Type":"ContainerStarted","Data":"abbf31db920a52c9f330dda4099afadcc293e363140e85acddf3c78f6e6bbd94"} Nov 02 20:52:51.099592 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:51.099530 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" podStartSLOduration=2.099516355 podStartE2EDuration="2.099516355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:52:51.098045566 +0000 UTC m=+3613.708697656" watchObservedRunningTime="2026-11-02 20:52:51.099516355 +0000 UTC m=+3613.710168445" Nov 02 20:52:51.838800 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:51.838767 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:51.843833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:51.843800 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:51.995662 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:51.995632 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:52:51.996695 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:52:51.996672 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:53:06.046941 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:06.046903 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:53:15.116354 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:15.116308 5399 generic.go:356] "Generic (PLEG): container finished" podID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerID="be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a" exitCode=0 Nov 02 20:53:15.116819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:15.116382 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" event={"ID":"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe","Type":"ContainerDied","Data":"be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a"} Nov 02 20:53:15.116819 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:15.116795 5399 scope.go:122] "RemoveContainer" containerID="be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a" Nov 02 20:53:15.118098 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:15.118042 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:53:15.118155 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:15.118130 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:53:15.118194 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:15.118176 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:53:16.124313 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:16.124259 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" event={"ID":"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe","Type":"ContainerStarted","Data":"d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06"} Nov 02 20:53:16.845503 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:16.845463 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:53:16.846370 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:16.846330 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 20:53:16.846585 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:16.846554 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 20:53:17.846305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:17.841755 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 20:53:17.846305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:17.841822 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 20:53:18.258995 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:53:18.258900 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:53:18.848302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:18.844746 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 20:53:18.848302 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:18.844816 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 20:53:19.846762 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:19.846718 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 20:53:19.847056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:19.847026 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 20:53:20.840736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:20.840693 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Nov 02 20:53:20.841995 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:20.841959 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Nov 02 20:53:21.843284 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:21.843249 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:53:21.843753 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:21.843598 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:53:21.844351 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:21.844331 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:53:27.134437 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:27.134398 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:53:48.236728 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:53:48.236680 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:53:52.788861 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:53:52.788819 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:54:06.909326 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:06.909270 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb"] Nov 02 20:54:06.912851 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:06.912834 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:06.962096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:06.962052 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:54:06.962096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:06.962083 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:54:06.969704 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:06.969665 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78ff6c14-7d2nb\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:06.969881 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:06.969825 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd6z2\" (UniqueName: \"kubernetes.io/projected/fc5929b8-90f7-4578-a6be-4aa053addc44-kube-api-access-cd6z2\") pod \"maas-api-revoke-sub-78ff6c14-7d2nb\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:06.969881 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:06.969871 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78ff6c14-7d2nb\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:07.008270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.008233 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:54:07.008270 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.008252 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:54:07.010503 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.010472 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb"] Nov 02 20:54:07.071113 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.071077 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78ff6c14-7d2nb\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:07.071315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.071180 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd6z2\" (UniqueName: \"kubernetes.io/projected/fc5929b8-90f7-4578-a6be-4aa053addc44-kube-api-access-cd6z2\") pod \"maas-api-revoke-sub-78ff6c14-7d2nb\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:07.071315 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.071215 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78ff6c14-7d2nb\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:07.071949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.071911 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78ff6c14-7d2nb\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:07.073486 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.073463 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78ff6c14-7d2nb\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:07.124368 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.124334 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd6z2\" (UniqueName: \"kubernetes.io/projected/fc5929b8-90f7-4578-a6be-4aa053addc44-kube-api-access-cd6z2\") pod \"maas-api-revoke-sub-78ff6c14-7d2nb\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:07.226045 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.225958 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:07.592505 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:07.592473 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb"] Nov 02 20:54:07.594502 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:54:07.594476 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5929b8_90f7_4578_a6be_4aa053addc44.slice/crio-8b76580be08655e520dc08b4630a345952a3189ec6753b2ecf526fb9ded88b7f WatchSource:0}: Error finding container 8b76580be08655e520dc08b4630a345952a3189ec6753b2ecf526fb9ded88b7f: Status 404 returned error can't find the container with id 8b76580be08655e520dc08b4630a345952a3189ec6753b2ecf526fb9ded88b7f Nov 02 20:54:08.420547 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:08.420514 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" event={"ID":"fc5929b8-90f7-4578-a6be-4aa053addc44","Type":"ContainerStarted","Data":"3f9c0e332331315dbb3b8dd3d42cb32a9068477296c59f2478c8d69119b85f7d"} Nov 02 20:54:08.421041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:08.420573 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" event={"ID":"fc5929b8-90f7-4578-a6be-4aa053addc44","Type":"ContainerStarted","Data":"8b76580be08655e520dc08b4630a345952a3189ec6753b2ecf526fb9ded88b7f"} Nov 02 20:54:08.526119 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:08.526066 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" podStartSLOduration=2.5260521000000002 podStartE2EDuration="2.5260521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:08.524994606 +0000 UTC m=+3691.135646697" watchObservedRunningTime="2026-11-02 20:54:08.5260521 +0000 UTC m=+3691.136704191" Nov 02 20:54:09.428144 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:09.428105 5399 generic.go:356] "Generic (PLEG): container finished" podID="fc5929b8-90f7-4578-a6be-4aa053addc44" containerID="3f9c0e332331315dbb3b8dd3d42cb32a9068477296c59f2478c8d69119b85f7d" exitCode=0 Nov 02 20:54:09.428590 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:09.428188 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" event={"ID":"fc5929b8-90f7-4578-a6be-4aa053addc44","Type":"ContainerDied","Data":"3f9c0e332331315dbb3b8dd3d42cb32a9068477296c59f2478c8d69119b85f7d"} Nov 02 20:54:10.575165 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:10.575134 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:10.708642 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:10.708562 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-xks-ca\") pod \"fc5929b8-90f7-4578-a6be-4aa053addc44\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " Nov 02 20:54:10.708812 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:10.708672 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-openshift-ca\") pod \"fc5929b8-90f7-4578-a6be-4aa053addc44\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " Nov 02 20:54:10.708812 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:10.708780 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5929b8-90f7-4578-a6be-4aa053addc44-kube-api-access-cd6z2\" (UniqueName: \"kubernetes.io/projected/fc5929b8-90f7-4578-a6be-4aa053addc44-kube-api-access-cd6z2\") pod \"fc5929b8-90f7-4578-a6be-4aa053addc44\" (UID: \"fc5929b8-90f7-4578-a6be-4aa053addc44\") " Nov 02 20:54:10.709056 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:10.709025 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-openshift-ca" pod "fc5929b8-90f7-4578-a6be-4aa053addc44" (UID: "fc5929b8-90f7-4578-a6be-4aa053addc44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:10.709196 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:10.709181 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:10.710826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:10.710799 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5929b8-90f7-4578-a6be-4aa053addc44-kube-api-access-cd6z2" pod "fc5929b8-90f7-4578-a6be-4aa053addc44" (UID: "fc5929b8-90f7-4578-a6be-4aa053addc44"). InnerVolumeSpecName "kube-api-access-cd6z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:10.710954 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:10.710840 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-xks-ca" pod "fc5929b8-90f7-4578-a6be-4aa053addc44" (UID: "fc5929b8-90f7-4578-a6be-4aa053addc44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:54:10.810775 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:10.810733 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd6z2\" (UniqueName: \"kubernetes.io/projected/fc5929b8-90f7-4578-a6be-4aa053addc44-kube-api-access-cd6z2\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:10.810775 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:10.810770 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5929b8-90f7-4578-a6be-4aa053addc44-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:11.440586 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:11.440558 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" Nov 02 20:54:11.440754 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:11.440555 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb" event={"ID":"fc5929b8-90f7-4578-a6be-4aa053addc44","Type":"ContainerDied","Data":"8b76580be08655e520dc08b4630a345952a3189ec6753b2ecf526fb9ded88b7f"} Nov 02 20:54:11.440754 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:11.440679 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b76580be08655e520dc08b4630a345952a3189ec6753b2ecf526fb9ded88b7f" Nov 02 20:54:18.236505 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:54:18.236424 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:54:21.759722 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:21.759682 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:54:29.038255 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.038214 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9"] Nov 02 20:54:29.038890 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.038867 5399 cpu_manager.go:414] "containerMap: removing container" podUID="fc5929b8-90f7-4578-a6be-4aa053addc44" containerName="revoke-keys" Nov 02 20:54:29.039002 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.038893 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5929b8-90f7-4578-a6be-4aa053addc44" containerName="revoke-keys" Nov 02 20:54:29.039062 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.039043 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5929b8-90f7-4578-a6be-4aa053addc44" containerName="revoke-keys" Nov 02 20:54:29.042403 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.042382 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.137709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.137679 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:54:29.137709 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.137683 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:54:29.137968 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.137822 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:54:29.137968 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.137930 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:54:29.140059 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.140037 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9"] Nov 02 20:54:29.200318 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.200287 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s6cd\" (UniqueName: \"kubernetes.io/projected/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-kube-api-access-8s6cd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.200522 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.200347 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.200522 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.200428 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.301483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.301396 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6cd\" (UniqueName: \"kubernetes.io/projected/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-kube-api-access-8s6cd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.301483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.301453 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.301483 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.301474 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.302067 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.302044 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.303857 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.303831 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.354348 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.354307 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6cd\" (UniqueName: \"kubernetes.io/projected/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-kube-api-access-8s6cd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.356074 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.356051 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:29.718787 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:29.718758 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9"] Nov 02 20:54:29.720888 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:54:29.720861 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03db14a_e5d7_4cc4_b5e7_8e289ca04763.slice/crio-23c9b63c4e831315830f080c1ffb588ef3458bb148fddbc6d0aa34588ac7cef2 WatchSource:0}: Error finding container 23c9b63c4e831315830f080c1ffb588ef3458bb148fddbc6d0aa34588ac7cef2: Status 404 returned error can't find the container with id 23c9b63c4e831315830f080c1ffb588ef3458bb148fddbc6d0aa34588ac7cef2 Nov 02 20:54:30.544736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:30.544701 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" event={"ID":"b03db14a-e5d7-4cc4-b5e7-8e289ca04763","Type":"ContainerStarted","Data":"077429b47029720a1fc1f60c2ae2fcbe66d29106727bb7568394fde57373b242"} Nov 02 20:54:30.545241 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:30.544757 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" event={"ID":"b03db14a-e5d7-4cc4-b5e7-8e289ca04763","Type":"ContainerStarted","Data":"23c9b63c4e831315830f080c1ffb588ef3458bb148fddbc6d0aa34588ac7cef2"} Nov 02 20:54:30.651428 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:30.651371 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" podStartSLOduration=2.651357269 podStartE2EDuration="2.651357269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:30.649924119 +0000 UTC m=+3713.260576210" watchObservedRunningTime="2026-11-02 20:54:30.651357269 +0000 UTC m=+3713.262009353" Nov 02 20:54:31.551768 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:31.551731 5399 generic.go:356] "Generic (PLEG): container finished" podID="b03db14a-e5d7-4cc4-b5e7-8e289ca04763" containerID="077429b47029720a1fc1f60c2ae2fcbe66d29106727bb7568394fde57373b242" exitCode=0 Nov 02 20:54:31.552305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:31.551826 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" event={"ID":"b03db14a-e5d7-4cc4-b5e7-8e289ca04763","Type":"ContainerDied","Data":"077429b47029720a1fc1f60c2ae2fcbe66d29106727bb7568394fde57373b242"} Nov 02 20:54:32.699580 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:32.699557 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:32.835917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:32.835831 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-xks-ca\") pod \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " Nov 02 20:54:32.835917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:32.835874 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-kube-api-access-8s6cd\" (UniqueName: \"kubernetes.io/projected/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-kube-api-access-8s6cd\") pod \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " Nov 02 20:54:32.835917 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:32.835894 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-openshift-ca\") pod \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\" (UID: \"b03db14a-e5d7-4cc4-b5e7-8e289ca04763\") " Nov 02 20:54:32.836338 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:32.836305 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-openshift-ca" pod "b03db14a-e5d7-4cc4-b5e7-8e289ca04763" (UID: "b03db14a-e5d7-4cc4-b5e7-8e289ca04763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:32.838071 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:32.838038 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-xks-ca" pod "b03db14a-e5d7-4cc4-b5e7-8e289ca04763" (UID: "b03db14a-e5d7-4cc4-b5e7-8e289ca04763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:54:32.838169 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:32.838066 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-kube-api-access-8s6cd" pod "b03db14a-e5d7-4cc4-b5e7-8e289ca04763" (UID: "b03db14a-e5d7-4cc4-b5e7-8e289ca04763"). InnerVolumeSpecName "kube-api-access-8s6cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:32.937516 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:32.937479 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:32.937516 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:32.937511 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s6cd\" (UniqueName: \"kubernetes.io/projected/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-kube-api-access-8s6cd\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:32.937516 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:32.937521 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b03db14a-e5d7-4cc4-b5e7-8e289ca04763-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:33.563967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:33.563938 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" Nov 02 20:54:33.564153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:33.563930 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9" event={"ID":"b03db14a-e5d7-4cc4-b5e7-8e289ca04763","Type":"ContainerDied","Data":"23c9b63c4e831315830f080c1ffb588ef3458bb148fddbc6d0aa34588ac7cef2"} Nov 02 20:54:33.564153 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:33.564075 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c9b63c4e831315830f080c1ffb588ef3458bb148fddbc6d0aa34588ac7cef2" Nov 02 20:54:45.160239 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:45.160198 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:54:45.460865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:45.460779 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9"] Nov 02 20:54:45.507393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:45.507360 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0a02f7-c2sq9"] Nov 02 20:54:47.507464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:47.507431 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03db14a-e5d7-4cc4-b5e7-8e289ca04763" path="/var/lib/kubelet/pods/b03db14a-e5d7-4cc4-b5e7-8e289ca04763/volumes" Nov 02 20:54:48.236830 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:54:48.236788 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:54:51.980367 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:51.980331 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw"] Nov 02 20:54:51.980810 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:51.980545 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" containerID="cri-o://d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06" gracePeriod=30 Nov 02 20:54:53.609732 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.609691 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5"] Nov 02 20:54:53.610180 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.610164 5399 cpu_manager.go:414] "containerMap: removing container" podUID="b03db14a-e5d7-4cc4-b5e7-8e289ca04763" containerName="revoke-keys" Nov 02 20:54:53.610229 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.610182 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03db14a-e5d7-4cc4-b5e7-8e289ca04763" containerName="revoke-keys" Nov 02 20:54:53.610311 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.610267 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03db14a-e5d7-4cc4-b5e7-8e289ca04763" containerName="revoke-keys" Nov 02 20:54:53.613815 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.613795 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:53.667185 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.667145 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5"] Nov 02 20:54:53.710750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.710711 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:54:53.710750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.710741 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:54:53.710967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.710716 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:54:53.710967 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.710713 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:54:53.750952 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.750912 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wf5b5\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:53.751168 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.750971 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2wd\" (UniqueName: \"kubernetes.io/projected/84b21b29-f4f6-4677-8d8e-bd284e1fab40-kube-api-access-cm2wd\") pod \"maas-api-revoke-sub-3d4156f1-wf5b5\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:53.751168 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.751148 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wf5b5\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:53.852387 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.852347 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wf5b5\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:53.852602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.852410 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2wd\" (UniqueName: \"kubernetes.io/projected/84b21b29-f4f6-4677-8d8e-bd284e1fab40-kube-api-access-cm2wd\") pod \"maas-api-revoke-sub-3d4156f1-wf5b5\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:53.852602 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.852477 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wf5b5\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:53.853162 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.853140 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wf5b5\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:53.854801 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.854777 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wf5b5\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:53.905663 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.905585 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2wd\" (UniqueName: \"kubernetes.io/projected/84b21b29-f4f6-4677-8d8e-bd284e1fab40-kube-api-access-cm2wd\") pod \"maas-api-revoke-sub-3d4156f1-wf5b5\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:53.927624 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:53.927590 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:54.291501 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:54.290423 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5"] Nov 02 20:54:54.296716 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:54:54.296680 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b21b29_f4f6_4677_8d8e_bd284e1fab40.slice/crio-8a8b8898c17a5a032d6f033c5b00cd3fc37b803dfc1aa40bbb8615a5ba6aa3e2 WatchSource:0}: Error finding container 8a8b8898c17a5a032d6f033c5b00cd3fc37b803dfc1aa40bbb8615a5ba6aa3e2: Status 404 returned error can't find the container with id 8a8b8898c17a5a032d6f033c5b00cd3fc37b803dfc1aa40bbb8615a5ba6aa3e2 Nov 02 20:54:54.685564 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:54.685520 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" event={"ID":"84b21b29-f4f6-4677-8d8e-bd284e1fab40","Type":"ContainerStarted","Data":"bf36e48abf08f706295cf6c9b81793548c19a793c028ad5f7dc0c7d73e64e01e"} Nov 02 20:54:54.685983 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:54.685585 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" event={"ID":"84b21b29-f4f6-4677-8d8e-bd284e1fab40","Type":"ContainerStarted","Data":"8a8b8898c17a5a032d6f033c5b00cd3fc37b803dfc1aa40bbb8615a5ba6aa3e2"} Nov 02 20:54:54.791696 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:54.791644 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" podStartSLOduration=1.7916313499999998 podStartE2EDuration="1.79163135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:54.790512022 +0000 UTC m=+3737.401164115" watchObservedRunningTime="2026-11-02 20:54:54.79163135 +0000 UTC m=+3737.402283495" Nov 02 20:54:55.691638 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:55.691600 5399 generic.go:356] "Generic (PLEG): container finished" podID="84b21b29-f4f6-4677-8d8e-bd284e1fab40" containerID="bf36e48abf08f706295cf6c9b81793548c19a793c028ad5f7dc0c7d73e64e01e" exitCode=0 Nov 02 20:54:55.692041 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:55.691684 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" event={"ID":"84b21b29-f4f6-4677-8d8e-bd284e1fab40","Type":"ContainerDied","Data":"bf36e48abf08f706295cf6c9b81793548c19a793c028ad5f7dc0c7d73e64e01e"} Nov 02 20:54:56.836487 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:56.836461 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:56.985253 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:56.985164 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b21b29-f4f6-4677-8d8e-bd284e1fab40-kube-api-access-cm2wd\" (UniqueName: \"kubernetes.io/projected/84b21b29-f4f6-4677-8d8e-bd284e1fab40-kube-api-access-cm2wd\") pod \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " Nov 02 20:54:56.985253 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:56.985214 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-xks-ca\") pod \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " Nov 02 20:54:56.985536 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:56.985351 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-openshift-ca\") pod \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\" (UID: \"84b21b29-f4f6-4677-8d8e-bd284e1fab40\") " Nov 02 20:54:56.985727 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:56.985702 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-openshift-ca" pod "84b21b29-f4f6-4677-8d8e-bd284e1fab40" (UID: "84b21b29-f4f6-4677-8d8e-bd284e1fab40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:56.987373 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:56.987346 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-xks-ca" pod "84b21b29-f4f6-4677-8d8e-bd284e1fab40" (UID: "84b21b29-f4f6-4677-8d8e-bd284e1fab40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:54:56.987463 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:56.987440 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b21b29-f4f6-4677-8d8e-bd284e1fab40-kube-api-access-cm2wd" pod "84b21b29-f4f6-4677-8d8e-bd284e1fab40" (UID: "84b21b29-f4f6-4677-8d8e-bd284e1fab40"). InnerVolumeSpecName "kube-api-access-cm2wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:57.086756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.086716 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm2wd\" (UniqueName: \"kubernetes.io/projected/84b21b29-f4f6-4677-8d8e-bd284e1fab40-kube-api-access-cm2wd\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.086756 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.086759 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.086971 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.086777 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b21b29-f4f6-4677-8d8e-bd284e1fab40-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.221070 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.221043 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:54:57.389103 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389069 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-envoy\") pod \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " Nov 02 20:54:57.389103 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389111 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-data\") pod \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " Nov 02 20:54:57.389360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389134 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-credential-socket\") pod \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " Nov 02 20:54:57.389360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389206 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-podinfo\") pod \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " Nov 02 20:54:57.389360 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389247 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istiod-ca-cert\") pod \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " Nov 02 20:54:57.389540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389361 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-token\" (UniqueName: \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-token\") pod \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " Nov 02 20:54:57.389540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389385 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-certs\") pod \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " Nov 02 20:54:57.389540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389411 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-data" pod "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" (UID: "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:57.389540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389435 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-socket\") pod \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " Nov 02 20:54:57.389540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389463 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-kube-api-access-zl9gg\" (UniqueName: \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-kube-api-access-zl9gg\") pod \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\" (UID: \"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe\") " Nov 02 20:54:57.389540 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389513 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-credential-socket" pod "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" (UID: "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:57.389862 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389697 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-certs" pod "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" (UID: "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:57.389862 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389709 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-socket" pod "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" (UID: "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:57.389972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389905 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.389972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389925 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.389972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389941 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.389972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389954 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.389972 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.389923 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istiod-ca-cert" pod "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" (UID: "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:57.391764 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.391743 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-envoy" pod "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" (UID: "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:54:57.391907 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.391883 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-kube-api-access-zl9gg" pod "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" (UID: "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe"). InnerVolumeSpecName "kube-api-access-zl9gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:57.391990 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.391970 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-token" pod "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" (UID: "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:57.392053 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.391970 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-podinfo" pod "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" (UID: "e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:54:57.491541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.491505 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.491541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.491536 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.491541 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.491548 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl9gg\" (UniqueName: \"kubernetes.io/projected/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-kube-api-access-zl9gg\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.491788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.491557 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.491788 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.491566 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:57.702743 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.702660 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" Nov 02 20:54:57.702910 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.702659 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5" event={"ID":"84b21b29-f4f6-4677-8d8e-bd284e1fab40","Type":"ContainerDied","Data":"8a8b8898c17a5a032d6f033c5b00cd3fc37b803dfc1aa40bbb8615a5ba6aa3e2"} Nov 02 20:54:57.702910 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.702793 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a8b8898c17a5a032d6f033c5b00cd3fc37b803dfc1aa40bbb8615a5ba6aa3e2" Nov 02 20:54:57.704461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.704413 5399 generic.go:356] "Generic (PLEG): container finished" podID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerID="d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06" exitCode=0 Nov 02 20:54:57.704461 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.704445 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" event={"ID":"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe","Type":"ContainerDied","Data":"d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06"} Nov 02 20:54:57.704865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.704488 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" Nov 02 20:54:57.704865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.704502 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw" event={"ID":"e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe","Type":"ContainerDied","Data":"abbf31db920a52c9f330dda4099afadcc293e363140e85acddf3c78f6e6bbd94"} Nov 02 20:54:57.704865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.704536 5399 scope.go:122] "RemoveContainer" containerID="d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06" Nov 02 20:54:57.713842 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.713816 5399 scope.go:122] "RemoveContainer" containerID="be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a" Nov 02 20:54:57.722100 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.722081 5399 scope.go:122] "RemoveContainer" containerID="d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06" Nov 02 20:54:57.722401 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:54:57.722379 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06\": container with ID starting with d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06 not found: ID does not exist" containerID="d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06" Nov 02 20:54:57.722448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.722410 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06"} err="failed to get container status \"d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06\": rpc error: code = NotFound desc = could not find container \"d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06\": container with ID starting with d48cfc39374891a1f10f477ea4637a95e5dde1f6b5fc11a8775679453d99ee06 not found: ID does not exist" Nov 02 20:54:57.722448 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.722429 5399 scope.go:122] "RemoveContainer" containerID="be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a" Nov 02 20:54:57.722701 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:54:57.722686 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a\": container with ID starting with be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a not found: ID does not exist" containerID="be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a" Nov 02 20:54:57.722750 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.722705 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a"} err="failed to get container status \"be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a\": rpc error: code = NotFound desc = could not find container \"be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a\": container with ID starting with be16dd2c88d049f9f5d216165783a6589a82f87ecac0c264f9846c7a4336433a not found: ID does not exist" Nov 02 20:54:57.857841 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.857803 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw"] Nov 02 20:54:57.870580 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:57.870548 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0a02f7-openshift-default-dc9cd5f7-pkcjw"] Nov 02 20:54:59.506593 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:54:59.506516 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" path="/var/lib/kubelet/pods/e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe/volumes" Nov 02 20:55:08.156183 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:55:08.156150 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:55:18.236306 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:55:18.236245 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:55:18.940795 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:55:18.940755 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:55:48.236452 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:55:48.236364 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:56:18.235687 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:56:18.235647 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:56:48.236406 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:56:48.236361 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:56:51.846094 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:56:51.846058 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:57:08.944948 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:08.944911 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:57:16.667577 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:16.667537 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:57:18.235793 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:57:18.235755 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:57:37.828763 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.828716 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:57:37.891118 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.891077 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28"] Nov 02 20:57:37.891587 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.891568 5399 cpu_manager.go:414] "containerMap: removing container" podUID="84b21b29-f4f6-4677-8d8e-bd284e1fab40" containerName="revoke-keys" Nov 02 20:57:37.891694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.891588 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b21b29-f4f6-4677-8d8e-bd284e1fab40" containerName="revoke-keys" Nov 02 20:57:37.891694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.891633 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" Nov 02 20:57:37.891694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.891643 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" Nov 02 20:57:37.891694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.891660 5399 cpu_manager.go:414] "containerMap: removing container" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" Nov 02 20:57:37.891694 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.891668 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" Nov 02 20:57:37.891961 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.891791 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" Nov 02 20:57:37.891961 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.891807 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e436bf1e-d9d5-4bb1-a4e1-f1cf5a607bbe" containerName="istio-proxy" Nov 02 20:57:37.891961 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.891819 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b21b29-f4f6-4677-8d8e-bd284e1fab40" containerName="revoke-keys" Nov 02 20:57:37.894998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.894973 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:37.947813 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.947781 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-d512fb-openshift-default-dockercfg-jcs57\"" Nov 02 20:57:37.992502 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:37.992469 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28"] Nov 02 20:57:38.055423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.055385 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-credential-socket\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.055423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.055422 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-envoy\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.055679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.055450 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-istio-token\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.055679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.055607 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-certs\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.055679 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.055675 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c853e579-0cc2-483e-a74a-ab19f49db767-istiod-ca-cert\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.055854 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.055771 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-socket\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.055854 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.055808 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-data\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.055945 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.055910 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c853e579-0cc2-483e-a74a-ab19f49db767-istio-podinfo\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.055998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.055945 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kgs\" (UniqueName: \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-kube-api-access-t8kgs\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.157482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.157381 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c853e579-0cc2-483e-a74a-ab19f49db767-istiod-ca-cert\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.157482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.157454 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-socket\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.157482 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.157477 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-data\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.157998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.157544 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c853e579-0cc2-483e-a74a-ab19f49db767-istio-podinfo\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.157998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.157581 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kgs\" (UniqueName: \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-kube-api-access-t8kgs\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.157998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.157628 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-credential-socket\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.157998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.157650 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-envoy\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.157998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.157686 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-istio-token\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.157998 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.157737 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-certs\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.158353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.158095 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-certs\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.158353 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.158189 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c853e579-0cc2-483e-a74a-ab19f49db767-istiod-ca-cert\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.158470 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.158431 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-credential-socket\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.159480 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.158619 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-data\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.159480 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.159239 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-socket\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.161087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.161043 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-envoy\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.162061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.162036 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c853e579-0cc2-483e-a74a-ab19f49db767-istio-podinfo\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.212214 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.212171 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kgs\" (UniqueName: \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-kube-api-access-t8kgs\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.212420 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.212329 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-istio-token\") pod \"e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.507598 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.507500 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:38.917329 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.917286 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28"] Nov 02 20:57:38.955040 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:57:38.955011 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc853e579_0cc2_483e_a74a_ab19f49db767.slice/crio-1d36149d23ed5e875b052d03e98e2a1c1dba3c763bcacf1bf0d971e8b130337a WatchSource:0}: Error finding container 1d36149d23ed5e875b052d03e98e2a1c1dba3c763bcacf1bf0d971e8b130337a: Status 404 returned error can't find the container with id 1d36149d23ed5e875b052d03e98e2a1c1dba3c763bcacf1bf0d971e8b130337a Nov 02 20:57:38.957330 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.957286 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:57:38.957410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.957371 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:57:38.957410 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:38.957400 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:57:39.622561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:39.622525 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" event={"ID":"c853e579-0cc2-483e-a74a-ab19f49db767","Type":"ContainerStarted","Data":"b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef"} Nov 02 20:57:39.622807 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:39.622579 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" event={"ID":"c853e579-0cc2-483e-a74a-ab19f49db767","Type":"ContainerStarted","Data":"1d36149d23ed5e875b052d03e98e2a1c1dba3c763bcacf1bf0d971e8b130337a"} Nov 02 20:57:39.823293 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:39.823231 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" podStartSLOduration=2.823216045 podStartE2EDuration="2.823216045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:39.775968299 +0000 UTC m=+3902.386620390" watchObservedRunningTime="2026-11-02 20:57:39.823216045 +0000 UTC m=+3902.433868135" Nov 02 20:57:40.507964 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:40.507926 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:40.513033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:40.513007 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:40.627766 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:40.627733 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:40.628752 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:40.628732 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:57:48.237360 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:57:48.237315 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:57:56.162201 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:57:56.162156 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:58:04.779317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:04.779252 5399 generic.go:356] "Generic (PLEG): container finished" podID="c853e579-0cc2-483e-a74a-ab19f49db767" containerID="b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef" exitCode=0 Nov 02 20:58:04.779783 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:04.779322 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" event={"ID":"c853e579-0cc2-483e-a74a-ab19f49db767","Type":"ContainerDied","Data":"b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef"} Nov 02 20:58:04.779783 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:04.779699 5399 scope.go:122] "RemoveContainer" containerID="b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef" Nov 02 20:58:04.780918 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:04.780890 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:58:04.780986 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:04.780961 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:58:04.781033 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:04.780995 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:58:05.786760 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:05.786724 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" event={"ID":"c853e579-0cc2-483e-a74a-ab19f49db767","Type":"ContainerStarted","Data":"47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f"} Nov 02 20:58:06.507949 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:06.507915 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:58:06.508736 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:06.508714 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 20:58:06.508849 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:06.508758 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 20:58:07.508733 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:07.508699 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 20:58:07.509148 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:07.508746 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 20:58:08.507855 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:08.507815 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 20:58:08.508063 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:08.507867 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 20:58:09.508676 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:09.508642 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 20:58:09.509190 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:09.508701 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 20:58:10.508068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:10.508028 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Nov 02 20:58:10.508321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:10.508108 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Nov 02 20:58:10.630068 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:10.630017 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:58:11.511927 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:11.511898 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:58:11.512918 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:11.512901 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:58:18.237804 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:58:18.237763 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:58:25.260009 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:25.259972 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:58:48.235639 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:58:48.235551 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:58:50.920734 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:58:50.920697 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:59:06.108439 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.108406 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2"] Nov 02 20:59:06.112200 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.112183 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.158865 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.158824 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v9bs\" (UniqueName: \"kubernetes.io/projected/d5731ea0-9ec6-4e21-97ba-4b45d3513952-kube-api-access-5v9bs\") pod \"maas-api-revoke-sub-57c93993-fpkp2\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.159029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.158885 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57c93993-fpkp2\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.159029 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.158906 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57c93993-fpkp2\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.165089 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.165063 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2"] Nov 02 20:59:06.211690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.211665 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:59:06.211690 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.211690 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:59:06.213880 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.213860 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:59:06.213974 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.213866 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:59:06.259660 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.259628 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v9bs\" (UniqueName: \"kubernetes.io/projected/d5731ea0-9ec6-4e21-97ba-4b45d3513952-kube-api-access-5v9bs\") pod \"maas-api-revoke-sub-57c93993-fpkp2\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.259827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.259675 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57c93993-fpkp2\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.259827 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.259692 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57c93993-fpkp2\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.260300 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.260256 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57c93993-fpkp2\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.262004 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.261977 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57c93993-fpkp2\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.316393 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.316365 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v9bs\" (UniqueName: \"kubernetes.io/projected/d5731ea0-9ec6-4e21-97ba-4b45d3513952-kube-api-access-5v9bs\") pod \"maas-api-revoke-sub-57c93993-fpkp2\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.425309 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.425215 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:06.787654 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:59:06.787606 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5731ea0_9ec6_4e21_97ba_4b45d3513952.slice/crio-0f7dbaac8aa22696f177715903455eed5c2b2e563fc63994e0a60b92c392bed9 WatchSource:0}: Error finding container 0f7dbaac8aa22696f177715903455eed5c2b2e563fc63994e0a60b92c392bed9: Status 404 returned error can't find the container with id 0f7dbaac8aa22696f177715903455eed5c2b2e563fc63994e0a60b92c392bed9 Nov 02 20:59:06.789454 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:06.789424 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2"] Nov 02 20:59:07.115659 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:07.115619 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" event={"ID":"d5731ea0-9ec6-4e21-97ba-4b45d3513952","Type":"ContainerStarted","Data":"469b8c33c4a5442a98ae07fc657902c902971920e445fbcab6264e3045b31bc0"} Nov 02 20:59:07.116108 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:07.115678 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" event={"ID":"d5731ea0-9ec6-4e21-97ba-4b45d3513952","Type":"ContainerStarted","Data":"0f7dbaac8aa22696f177715903455eed5c2b2e563fc63994e0a60b92c392bed9"} Nov 02 20:59:07.230067 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:07.230011 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" podStartSLOduration=1.22999708 podStartE2EDuration="1.22999708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:59:07.227971415 +0000 UTC m=+3989.838623508" watchObservedRunningTime="2026-11-02 20:59:07.22999708 +0000 UTC m=+3989.840649172" Nov 02 20:59:08.122559 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:08.122521 5399 generic.go:356] "Generic (PLEG): container finished" podID="d5731ea0-9ec6-4e21-97ba-4b45d3513952" containerID="469b8c33c4a5442a98ae07fc657902c902971920e445fbcab6264e3045b31bc0" exitCode=0 Nov 02 20:59:08.122980 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:08.122583 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" event={"ID":"d5731ea0-9ec6-4e21-97ba-4b45d3513952","Type":"ContainerDied","Data":"469b8c33c4a5442a98ae07fc657902c902971920e445fbcab6264e3045b31bc0"} Nov 02 20:59:09.266726 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:09.266699 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:09.292160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:09.292127 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-openshift-ca\") pod \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " Nov 02 20:59:09.292160 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:09.292162 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-xks-ca\") pod \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " Nov 02 20:59:09.292422 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:09.292200 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5731ea0-9ec6-4e21-97ba-4b45d3513952-kube-api-access-5v9bs\" (UniqueName: \"kubernetes.io/projected/d5731ea0-9ec6-4e21-97ba-4b45d3513952-kube-api-access-5v9bs\") pod \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\" (UID: \"d5731ea0-9ec6-4e21-97ba-4b45d3513952\") " Nov 02 20:59:09.292553 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:09.292526 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-openshift-ca" pod "d5731ea0-9ec6-4e21-97ba-4b45d3513952" (UID: "d5731ea0-9ec6-4e21-97ba-4b45d3513952"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:59:09.294372 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:09.294348 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-xks-ca" pod "d5731ea0-9ec6-4e21-97ba-4b45d3513952" (UID: "d5731ea0-9ec6-4e21-97ba-4b45d3513952"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:59:09.294463 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:09.294395 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5731ea0-9ec6-4e21-97ba-4b45d3513952-kube-api-access-5v9bs" pod "d5731ea0-9ec6-4e21-97ba-4b45d3513952" (UID: "d5731ea0-9ec6-4e21-97ba-4b45d3513952"). InnerVolumeSpecName "kube-api-access-5v9bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:59:09.393891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:09.393816 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v9bs\" (UniqueName: \"kubernetes.io/projected/d5731ea0-9ec6-4e21-97ba-4b45d3513952-kube-api-access-5v9bs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:09.393891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:09.393844 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:09.393891 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:09.393856 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5731ea0-9ec6-4e21-97ba-4b45d3513952-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:10.134105 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:10.134072 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" Nov 02 20:59:10.134321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:10.134103 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2" event={"ID":"d5731ea0-9ec6-4e21-97ba-4b45d3513952","Type":"ContainerDied","Data":"0f7dbaac8aa22696f177715903455eed5c2b2e563fc63994e0a60b92c392bed9"} Nov 02 20:59:10.134321 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:10.134163 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f7dbaac8aa22696f177715903455eed5c2b2e563fc63994e0a60b92c392bed9" Nov 02 20:59:11.052535 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:11.052504 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb"] Nov 02 20:59:11.102193 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:11.102161 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ff6c14-7d2nb"] Nov 02 20:59:11.506298 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:11.506251 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5929b8-90f7-4578-a6be-4aa053addc44" path="/var/lib/kubelet/pods/fc5929b8-90f7-4578-a6be-4aa053addc44/volumes" Nov 02 20:59:17.657782 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.657746 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75"] Nov 02 20:59:17.658189 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.658178 5399 cpu_manager.go:414] "containerMap: removing container" podUID="d5731ea0-9ec6-4e21-97ba-4b45d3513952" containerName="revoke-keys" Nov 02 20:59:17.658233 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.658190 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5731ea0-9ec6-4e21-97ba-4b45d3513952" containerName="revoke-keys" Nov 02 20:59:17.658317 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.658306 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5731ea0-9ec6-4e21-97ba-4b45d3513952" containerName="revoke-keys" Nov 02 20:59:17.661384 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.661367 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:17.714116 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.714076 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75"] Nov 02 20:59:17.760963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.760928 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:59:17.760963 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.760953 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:59:17.761202 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.760998 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:59:17.761202 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.761079 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cx6dd\"" Nov 02 20:59:17.786172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.786136 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:17.786172 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.786172 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:17.786444 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.786320 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgk48\" (UniqueName: \"kubernetes.io/projected/871ac95a-334f-48df-81e6-117a87908ac2-kube-api-access-qgk48\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:17.887542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.887502 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:17.887542 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.887541 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:17.887833 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.887601 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgk48\" (UniqueName: \"kubernetes.io/projected/871ac95a-334f-48df-81e6-117a87908ac2-kube-api-access-qgk48\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:17.888181 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.888158 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:17.890039 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.890007 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:17.944411 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.944333 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgk48\" (UniqueName: \"kubernetes.io/projected/871ac95a-334f-48df-81e6-117a87908ac2-kube-api-access-qgk48\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:17.974472 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:17.974430 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:18.237385 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:59:18.237301 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:59:18.342378 ip-10-0-130-111 kubenswrapper[5399]: W1102 20:59:18.342339 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871ac95a_334f_48df_81e6_117a87908ac2.slice/crio-21a9fbe9b11ce905b26c26e90820c0cc67120ec86693575cfac9622c907e0a65 WatchSource:0}: Error finding container 21a9fbe9b11ce905b26c26e90820c0cc67120ec86693575cfac9622c907e0a65: Status 404 returned error can't find the container with id 21a9fbe9b11ce905b26c26e90820c0cc67120ec86693575cfac9622c907e0a65 Nov 02 20:59:18.344087 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:18.344009 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75"] Nov 02 20:59:18.948305 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:18.948254 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:59:19.183715 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:19.183679 5399 generic.go:356] "Generic (PLEG): container finished" podID="871ac95a-334f-48df-81e6-117a87908ac2" containerID="b999377020336e180e4307820f30da96f6fdfdb13cfc576a3a982ce3d98c356b" exitCode=0 Nov 02 20:59:19.183936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:19.183719 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" event={"ID":"871ac95a-334f-48df-81e6-117a87908ac2","Type":"ContainerDied","Data":"b999377020336e180e4307820f30da96f6fdfdb13cfc576a3a982ce3d98c356b"} Nov 02 20:59:19.183936 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:19.183776 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" event={"ID":"871ac95a-334f-48df-81e6-117a87908ac2","Type":"ContainerStarted","Data":"21a9fbe9b11ce905b26c26e90820c0cc67120ec86693575cfac9622c907e0a65"} Nov 02 20:59:20.326164 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:20.326141 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:20.412826 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:20.412796 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871ac95a-334f-48df-81e6-117a87908ac2-kube-api-access-qgk48\" (UniqueName: \"kubernetes.io/projected/871ac95a-334f-48df-81e6-117a87908ac2-kube-api-access-qgk48\") pod \"871ac95a-334f-48df-81e6-117a87908ac2\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " Nov 02 20:59:20.413020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:20.412872 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-openshift-ca\") pod \"871ac95a-334f-48df-81e6-117a87908ac2\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " Nov 02 20:59:20.413020 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:20.412917 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-xks-ca\") pod \"871ac95a-334f-48df-81e6-117a87908ac2\" (UID: \"871ac95a-334f-48df-81e6-117a87908ac2\") " Nov 02 20:59:20.413257 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:20.413233 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-openshift-ca" pod "871ac95a-334f-48df-81e6-117a87908ac2" (UID: "871ac95a-334f-48df-81e6-117a87908ac2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:59:20.413357 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:20.413340 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-openshift-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:20.414877 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:20.414853 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871ac95a-334f-48df-81e6-117a87908ac2-kube-api-access-qgk48" pod "871ac95a-334f-48df-81e6-117a87908ac2" (UID: "871ac95a-334f-48df-81e6-117a87908ac2"). InnerVolumeSpecName "kube-api-access-qgk48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:59:20.415129 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:20.415101 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-xks-ca" pod "871ac95a-334f-48df-81e6-117a87908ac2" (UID: "871ac95a-334f-48df-81e6-117a87908ac2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:59:20.514812 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:20.514729 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgk48\" (UniqueName: \"kubernetes.io/projected/871ac95a-334f-48df-81e6-117a87908ac2-kube-api-access-qgk48\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:20.514812 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:20.514762 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871ac95a-334f-48df-81e6-117a87908ac2-maas-api-xks-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:21.195301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:21.195253 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" event={"ID":"871ac95a-334f-48df-81e6-117a87908ac2","Type":"ContainerDied","Data":"21a9fbe9b11ce905b26c26e90820c0cc67120ec86693575cfac9622c907e0a65"} Nov 02 20:59:21.195301 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:21.195295 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75" Nov 02 20:59:21.195526 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:21.195325 5399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a9fbe9b11ce905b26c26e90820c0cc67120ec86693575cfac9622c907e0a65" Nov 02 20:59:33.790322 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:33.790266 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:59:34.249761 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:34.249725 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75"] Nov 02 20:59:34.299789 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:34.299755 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d512fb-tzs75"] Nov 02 20:59:35.506435 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:35.506399 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871ac95a-334f-48df-81e6-117a87908ac2" path="/var/lib/kubelet/pods/871ac95a-334f-48df-81e6-117a87908ac2/volumes" Nov 02 20:59:40.777961 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:40.777915 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28"] Nov 02 20:59:40.778512 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:40.778445 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" containerID="cri-o://47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f" gracePeriod=30 Nov 02 20:59:41.514246 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:41.514210 5399 patch_prober.go:29] interesting pod/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:59:41.514464 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:41.514293 5399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:59:46.034016 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.033987 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:59:46.072096 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072068 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-socket\") pod \"c853e579-0cc2-483e-a74a-ab19f49db767\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " Nov 02 20:59:46.072256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072108 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-data\") pod \"c853e579-0cc2-483e-a74a-ab19f49db767\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " Nov 02 20:59:46.072256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072128 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-certs\") pod \"c853e579-0cc2-483e-a74a-ab19f49db767\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " Nov 02 20:59:46.072256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072187 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c853e579-0cc2-483e-a74a-ab19f49db767-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c853e579-0cc2-483e-a74a-ab19f49db767-istio-podinfo\") pod \"c853e579-0cc2-483e-a74a-ab19f49db767\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " Nov 02 20:59:46.072256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072218 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-kube-api-access-t8kgs\" (UniqueName: \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-kube-api-access-t8kgs\") pod \"c853e579-0cc2-483e-a74a-ab19f49db767\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " Nov 02 20:59:46.072256 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072238 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-envoy\") pod \"c853e579-0cc2-483e-a74a-ab19f49db767\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " Nov 02 20:59:46.072561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072295 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c853e579-0cc2-483e-a74a-ab19f49db767-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c853e579-0cc2-483e-a74a-ab19f49db767-istiod-ca-cert\") pod \"c853e579-0cc2-483e-a74a-ab19f49db767\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " Nov 02 20:59:46.072561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072334 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-credential-socket\") pod \"c853e579-0cc2-483e-a74a-ab19f49db767\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " Nov 02 20:59:46.072561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072391 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-istio-token\" (UniqueName: \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-istio-token\") pod \"c853e579-0cc2-483e-a74a-ab19f49db767\" (UID: \"c853e579-0cc2-483e-a74a-ab19f49db767\") " Nov 02 20:59:46.072561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072388 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-data" pod "c853e579-0cc2-483e-a74a-ab19f49db767" (UID: "c853e579-0cc2-483e-a74a-ab19f49db767"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:46.072561 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072506 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-certs" pod "c853e579-0cc2-483e-a74a-ab19f49db767" (UID: "c853e579-0cc2-483e-a74a-ab19f49db767"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:46.072840 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072732 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-credential-socket" pod "c853e579-0cc2-483e-a74a-ab19f49db767" (UID: "c853e579-0cc2-483e-a74a-ab19f49db767"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:46.072840 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072751 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c853e579-0cc2-483e-a74a-ab19f49db767-istiod-ca-cert" pod "c853e579-0cc2-483e-a74a-ab19f49db767" (UID: "c853e579-0cc2-483e-a74a-ab19f49db767"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:59:46.072955 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072852 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-socket" pod "c853e579-0cc2-483e-a74a-ab19f49db767" (UID: "c853e579-0cc2-483e-a74a-ab19f49db767"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:46.072955 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072861 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:46.072955 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072881 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:46.072955 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.072896 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:46.074507 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.074481 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-envoy" pod "c853e579-0cc2-483e-a74a-ab19f49db767" (UID: "c853e579-0cc2-483e-a74a-ab19f49db767"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:46.074647 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.074517 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c853e579-0cc2-483e-a74a-ab19f49db767-istio-podinfo" pod "c853e579-0cc2-483e-a74a-ab19f49db767" (UID: "c853e579-0cc2-483e-a74a-ab19f49db767"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:59:46.074906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.074884 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-istio-token" pod "c853e579-0cc2-483e-a74a-ab19f49db767" (UID: "c853e579-0cc2-483e-a74a-ab19f49db767"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:59:46.074906 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.074895 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-kube-api-access-t8kgs" pod "c853e579-0cc2-483e-a74a-ab19f49db767" (UID: "c853e579-0cc2-483e-a74a-ab19f49db767"). InnerVolumeSpecName "kube-api-access-t8kgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:59:46.174423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.174387 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c853e579-0cc2-483e-a74a-ab19f49db767-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:46.174423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.174416 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:46.174423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.174426 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:46.174423 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.174435 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c853e579-0cc2-483e-a74a-ab19f49db767-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:46.174715 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.174444 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8kgs\" (UniqueName: \"kubernetes.io/projected/c853e579-0cc2-483e-a74a-ab19f49db767-kube-api-access-t8kgs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:46.174715 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.174455 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c853e579-0cc2-483e-a74a-ab19f49db767-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:46.341031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.340942 5399 generic.go:356] "Generic (PLEG): container finished" podID="c853e579-0cc2-483e-a74a-ab19f49db767" containerID="47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f" exitCode=0 Nov 02 20:59:46.341031 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.341000 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" Nov 02 20:59:46.341242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.341000 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" event={"ID":"c853e579-0cc2-483e-a74a-ab19f49db767","Type":"ContainerDied","Data":"47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f"} Nov 02 20:59:46.341242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.341132 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28" event={"ID":"c853e579-0cc2-483e-a74a-ab19f49db767","Type":"ContainerDied","Data":"1d36149d23ed5e875b052d03e98e2a1c1dba3c763bcacf1bf0d971e8b130337a"} Nov 02 20:59:46.341242 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.341163 5399 scope.go:122] "RemoveContainer" containerID="47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f" Nov 02 20:59:46.351427 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.351404 5399 scope.go:122] "RemoveContainer" containerID="b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef" Nov 02 20:59:46.359731 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.359710 5399 scope.go:122] "RemoveContainer" containerID="47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f" Nov 02 20:59:46.359994 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:59:46.359972 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f\": container with ID starting with 47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f not found: ID does not exist" containerID="47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f" Nov 02 20:59:46.360079 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.360008 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f"} err="failed to get container status \"47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f\": rpc error: code = NotFound desc = could not find container \"47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f\": container with ID starting with 47b5938d9b0969c80210995ad87fbab994571a38cc7ed901e0676247af92b54f not found: ID does not exist" Nov 02 20:59:46.360079 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.360033 5399 scope.go:122] "RemoveContainer" containerID="b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef" Nov 02 20:59:46.360314 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:59:46.360295 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef\": container with ID starting with b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef not found: ID does not exist" containerID="b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef" Nov 02 20:59:46.360366 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.360322 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef"} err="failed to get container status \"b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef\": rpc error: code = NotFound desc = could not find container \"b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef\": container with ID starting with b9fd340b4657f66d88a3ad95c4f6c5b01416754e7f3a925e85b1010138d9ebef not found: ID does not exist" Nov 02 20:59:46.505018 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.504982 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28"] Nov 02 20:59:46.507931 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:46.507909 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d512fb-openshift-default-6d7644c85-v5x28"] Nov 02 20:59:47.506722 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:47.506687 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" path="/var/lib/kubelet/pods/c853e579-0cc2-483e-a74a-ab19f49db767/volumes" Nov 02 20:59:48.236244 ip-10-0-130-111 kubenswrapper[5399]: E1102 20:59:48.236197 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 20:59:50.964648 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:50.964612 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 20:59:57.045495 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:57.045458 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5"] Nov 02 20:59:57.095297 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:57.095243 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wf5b5"] Nov 02 20:59:57.506061 ip-10-0-130-111 kubenswrapper[5399]: I1102 20:59:57.506024 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b21b29-f4f6-4677-8d8e-bd284e1fab40" path="/var/lib/kubelet/pods/84b21b29-f4f6-4677-8d8e-bd284e1fab40/volumes" Nov 02 21:00:18.237298 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:00:18.237198 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:00:34.653092 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:00:34.653060 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:00:39.776771 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:00:39.776738 5399 scope.go:122] "RemoveContainer" containerID="3f9c0e332331315dbb3b8dd3d42cb32a9068477296c59f2478c8d69119b85f7d" Nov 02 21:00:39.785689 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:00:39.785664 5399 scope.go:122] "RemoveContainer" containerID="077429b47029720a1fc1f60c2ae2fcbe66d29106727bb7568394fde57373b242" Nov 02 21:00:48.236260 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:00:48.236219 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:01:00.136969 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:01:00.136925 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:01:15.468512 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:01:15.468473 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:01:18.302192 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:01:18.302151 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:01:30.342801 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:01:30.338642 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:01:39.818409 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:01:39.818378 5399 scope.go:122] "RemoveContainer" containerID="bf36e48abf08f706295cf6c9b81793548c19a793c028ad5f7dc0c7d73e64e01e" Nov 02 21:01:41.429116 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:01:41.429077 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:01:48.236343 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:01:48.236250 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:02:08.445316 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.445244 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:02:08.872846 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.872804 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv"] Nov 02 21:02:08.873453 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.873436 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" Nov 02 21:02:08.873511 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.873456 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" Nov 02 21:02:08.873511 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.873475 5399 cpu_manager.go:414] "containerMap: removing container" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" Nov 02 21:02:08.873511 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.873483 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" Nov 02 21:02:08.873608 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.873513 5399 cpu_manager.go:414] "containerMap: removing container" podUID="871ac95a-334f-48df-81e6-117a87908ac2" containerName="revoke-keys" Nov 02 21:02:08.873608 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.873521 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871ac95a-334f-48df-81e6-117a87908ac2" containerName="revoke-keys" Nov 02 21:02:08.873675 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.873613 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="871ac95a-334f-48df-81e6-117a87908ac2" containerName="revoke-keys" Nov 02 21:02:08.873675 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.873630 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" Nov 02 21:02:08.877642 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.877612 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:08.929590 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.929549 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv"] Nov 02 21:02:08.963376 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.963343 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-metrics-certs\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:08.963572 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.963384 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-maas-api-tls\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:08.963572 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.963515 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8d5\" (UniqueName: \"kubernetes.io/projected/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-kube-api-access-vz8d5\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:08.963685 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.963653 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-openshift-service-ca\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:08.977520 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.977490 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:02:08.977520 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.977490 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:02:08.977767 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.977567 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 21:02:08.979590 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.979571 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vqtwv\"" Nov 02 21:02:08.979694 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:08.979607 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 21:02:09.064598 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.064558 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-openshift-service-ca\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:09.064827 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.064668 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-metrics-certs\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:09.064827 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.064702 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-maas-api-tls\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:09.064827 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.064768 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8d5\" (UniqueName: \"kubernetes.io/projected/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-kube-api-access-vz8d5\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:09.065412 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.065388 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-openshift-service-ca\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:09.067227 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.067204 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-metrics-certs\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:09.067339 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.067244 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-maas-api-tls\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:09.121532 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.121493 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8d5\" (UniqueName: \"kubernetes.io/projected/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-kube-api-access-vz8d5\") pod \"maas-api-556d9884fd-hkrfv\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:09.193328 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.193230 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:09.561860 ip-10-0-130-111 kubenswrapper[5399]: W1102 21:02:09.561830 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354b67e7_a8bc_49d9_9fac_dc33cfd0f559.slice/crio-70a06f6604d2012a694d0400d4d2357d7e65b9d752d67cd4d95888d53c213c33 WatchSource:0}: Error finding container 70a06f6604d2012a694d0400d4d2357d7e65b9d752d67cd4d95888d53c213c33: Status 404 returned error can't find the container with id 70a06f6604d2012a694d0400d4d2357d7e65b9d752d67cd4d95888d53c213c33 Nov 02 21:02:09.563302 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.563260 5399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:02:09.563457 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:09.563435 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv"] Nov 02 21:02:10.231813 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:10.231775 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" event={"ID":"354b67e7-a8bc-49d9-9fac-dc33cfd0f559","Type":"ContainerStarted","Data":"70a06f6604d2012a694d0400d4d2357d7e65b9d752d67cd4d95888d53c213c33"} Nov 02 21:02:11.239190 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:11.239152 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" event={"ID":"354b67e7-a8bc-49d9-9fac-dc33cfd0f559","Type":"ContainerStarted","Data":"959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c"} Nov 02 21:02:11.239718 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:11.239298 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:11.352783 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:11.352720 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" podStartSLOduration=2.4169802750000002 podStartE2EDuration="3.352705086s" totalImagesPullingTime="935.724811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:02:11.351414605 +0000 UTC m=+4173.962066695" watchObservedRunningTime="2026-11-02 21:02:11.352705086 +0000 UTC m=+4173.963357177" Nov 02 21:02:17.249835 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.249806 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:17.507368 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.507290 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv"] Nov 02 21:02:17.507544 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.507512 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" podUID="354b67e7-a8bc-49d9-9fac-dc33cfd0f559" containerName="maas-api" containerID="cri-o://959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c" gracePeriod=30 Nov 02 21:02:17.527590 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.527561 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2"] Nov 02 21:02:17.528109 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.528091 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c853e579-0cc2-483e-a74a-ab19f49db767" containerName="istio-proxy" Nov 02 21:02:17.531237 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.531217 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.549251 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.549224 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d374d07-7e73-4478-b647-41f4ed60550a-openshift-service-ca\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.549445 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.549427 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d374d07-7e73-4478-b647-41f4ed60550a-maas-api-tls\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.549515 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.549463 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d374d07-7e73-4478-b647-41f4ed60550a-metrics-certs\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.549607 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.549581 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvb7c\" (UniqueName: \"kubernetes.io/projected/6d374d07-7e73-4478-b647-41f4ed60550a-kube-api-access-lvb7c\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.585455 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.585426 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2"] Nov 02 21:02:17.650410 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.650377 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d374d07-7e73-4478-b647-41f4ed60550a-maas-api-tls\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.650577 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.650418 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d374d07-7e73-4478-b647-41f4ed60550a-metrics-certs\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.650577 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.650447 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvb7c\" (UniqueName: \"kubernetes.io/projected/6d374d07-7e73-4478-b647-41f4ed60550a-kube-api-access-lvb7c\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.650577 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.650489 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d374d07-7e73-4478-b647-41f4ed60550a-openshift-service-ca\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.651180 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.651154 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d374d07-7e73-4478-b647-41f4ed60550a-openshift-service-ca\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.652663 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.652644 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d374d07-7e73-4478-b647-41f4ed60550a-maas-api-tls\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.652893 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.652872 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d374d07-7e73-4478-b647-41f4ed60550a-metrics-certs\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.707044 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.707007 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvb7c\" (UniqueName: \"kubernetes.io/projected/6d374d07-7e73-4478-b647-41f4ed60550a-kube-api-access-lvb7c\") pod \"maas-api-5bc7c89d86-5dqt2\" (UID: \"6d374d07-7e73-4478-b647-41f4ed60550a\") " pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.755521 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.755496 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:17.852564 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.852531 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-metrics-certs\" (UniqueName: \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-metrics-certs\") pod \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " Nov 02 21:02:17.852564 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.852570 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-openshift-service-ca\") pod \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " Nov 02 21:02:17.852840 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.852608 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-kube-api-access-vz8d5\" (UniqueName: \"kubernetes.io/projected/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-kube-api-access-vz8d5\") pod \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " Nov 02 21:02:17.852840 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.852662 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-maas-api-tls\") pod \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\" (UID: \"354b67e7-a8bc-49d9-9fac-dc33cfd0f559\") " Nov 02 21:02:17.853029 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.853004 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-openshift-service-ca" pod "354b67e7-a8bc-49d9-9fac-dc33cfd0f559" (UID: "354b67e7-a8bc-49d9-9fac-dc33cfd0f559"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:02:17.854768 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.854743 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-metrics-certs" pod "354b67e7-a8bc-49d9-9fac-dc33cfd0f559" (UID: "354b67e7-a8bc-49d9-9fac-dc33cfd0f559"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:02:17.854865 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.854762 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-maas-api-tls" pod "354b67e7-a8bc-49d9-9fac-dc33cfd0f559" (UID: "354b67e7-a8bc-49d9-9fac-dc33cfd0f559"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:02:17.854865 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.854794 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-kube-api-access-vz8d5" pod "354b67e7-a8bc-49d9-9fac-dc33cfd0f559" (UID: "354b67e7-a8bc-49d9-9fac-dc33cfd0f559"). InnerVolumeSpecName "kube-api-access-vz8d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:17.892930 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.892890 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:17.954947 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.954908 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:17.955081 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.954954 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:17.955081 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.954971 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz8d5\" (UniqueName: \"kubernetes.io/projected/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-kube-api-access-vz8d5\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:17.955081 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:17.954987 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/354b67e7-a8bc-49d9-9fac-dc33cfd0f559-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:18.237582 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:02:18.237494 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:02:18.261642 ip-10-0-130-111 kubenswrapper[5399]: W1102 21:02:18.261608 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d374d07_7e73_4478_b647_41f4ed60550a.slice/crio-85642a4651e3eaa305fbc458f3fed016cbba90472689b8b24914878b9bff5928 WatchSource:0}: Error finding container 85642a4651e3eaa305fbc458f3fed016cbba90472689b8b24914878b9bff5928: Status 404 returned error can't find the container with id 85642a4651e3eaa305fbc458f3fed016cbba90472689b8b24914878b9bff5928 Nov 02 21:02:18.263544 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.263427 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2"] Nov 02 21:02:18.276292 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.276240 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" event={"ID":"6d374d07-7e73-4478-b647-41f4ed60550a","Type":"ContainerStarted","Data":"85642a4651e3eaa305fbc458f3fed016cbba90472689b8b24914878b9bff5928"} Nov 02 21:02:18.277525 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.277501 5399 generic.go:356] "Generic (PLEG): container finished" podID="354b67e7-a8bc-49d9-9fac-dc33cfd0f559" containerID="959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c" exitCode=0 Nov 02 21:02:18.277628 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.277568 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" Nov 02 21:02:18.277628 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.277580 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" event={"ID":"354b67e7-a8bc-49d9-9fac-dc33cfd0f559","Type":"ContainerDied","Data":"959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c"} Nov 02 21:02:18.277628 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.277615 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv" event={"ID":"354b67e7-a8bc-49d9-9fac-dc33cfd0f559","Type":"ContainerDied","Data":"70a06f6604d2012a694d0400d4d2357d7e65b9d752d67cd4d95888d53c213c33"} Nov 02 21:02:18.277800 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.277637 5399 scope.go:122] "RemoveContainer" containerID="959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c" Nov 02 21:02:18.286722 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.286706 5399 scope.go:122] "RemoveContainer" containerID="959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c" Nov 02 21:02:18.286983 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:02:18.286962 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c\": container with ID starting with 959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c not found: ID does not exist" containerID="959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c" Nov 02 21:02:18.287033 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.286991 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c"} err="failed to get container status \"959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c\": rpc error: code = NotFound desc = could not find container \"959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c\": container with ID starting with 959cb3f08246f9826cd448ddbfc1b9699eb1200d79b6ab16e63738d84c9d467c not found: ID does not exist" Nov 02 21:02:18.440439 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.440408 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv"] Nov 02 21:02:18.442320 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:18.442296 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-556d9884fd-hkrfv"] Nov 02 21:02:19.284381 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:19.284348 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" event={"ID":"6d374d07-7e73-4478-b647-41f4ed60550a","Type":"ContainerStarted","Data":"b4f53d572b80ddbe6dedf8c3fb541bed598ce61075b8fc5258e45b3a1b59c874"} Nov 02 21:02:19.284935 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:19.284439 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:19.397534 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:19.397440 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" podStartSLOduration=1.525293112 podStartE2EDuration="2.397426526s" totalImagesPullingTime="872.133414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:02:19.397029394 +0000 UTC m=+4182.007681496" watchObservedRunningTime="2026-11-02 21:02:19.397426526 +0000 UTC m=+4182.008078616" Nov 02 21:02:19.507158 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:19.507126 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="354b67e7-a8bc-49d9-9fac-dc33cfd0f559" path="/var/lib/kubelet/pods/354b67e7-a8bc-49d9-9fac-dc33cfd0f559/volumes" Nov 02 21:02:25.296600 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:25.296568 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bc7c89d86-5dqt2" Nov 02 21:02:32.908147 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:32.908107 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr"] Nov 02 21:02:32.908591 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:32.908539 5399 cpu_manager.go:414] "containerMap: removing container" podUID="354b67e7-a8bc-49d9-9fac-dc33cfd0f559" containerName="maas-api" Nov 02 21:02:32.908591 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:32.908551 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354b67e7-a8bc-49d9-9fac-dc33cfd0f559" containerName="maas-api" Nov 02 21:02:32.908679 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:32.908652 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="354b67e7-a8bc-49d9-9fac-dc33cfd0f559" containerName="maas-api" Nov 02 21:02:32.912166 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:32.912144 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:32.964195 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:32.964153 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3203ddac-serving-cert\"" Nov 02 21:02:33.002297 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.002245 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafaf75a-f79b-4c83-b57d-4a8a370f6519-openshift-service-ca\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.002493 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.002406 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng7vr\" (UniqueName: \"kubernetes.io/projected/fafaf75a-f79b-4c83-b57d-4a8a370f6519-kube-api-access-ng7vr\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.002493 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.002436 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-metrics-certs\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.002493 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.002460 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-maas-api-tls\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.016357 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.016328 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr"] Nov 02 21:02:33.103409 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.103378 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng7vr\" (UniqueName: \"kubernetes.io/projected/fafaf75a-f79b-4c83-b57d-4a8a370f6519-kube-api-access-ng7vr\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.103409 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.103412 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-metrics-certs\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.103626 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.103435 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-maas-api-tls\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.103626 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.103506 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafaf75a-f79b-4c83-b57d-4a8a370f6519-openshift-service-ca\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.103626 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:02:33.103593 5399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-serving-cert: secret "maas-api-e2e-ait-3203ddac-serving-cert" not found Nov 02 21:02:33.103738 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:02:33.103651 5399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-maas-api-tls podName:fafaf75a-f79b-4c83-b57d-4a8a370f6519 nodeName:}" failed. No retries permitted until 2026-11-02 21:02:33.603635078 +0000 UTC m=+4196.214287148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-maas-api-tls") pod "maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" (UID: "fafaf75a-f79b-4c83-b57d-4a8a370f6519") : secret "maas-api-e2e-ait-3203ddac-serving-cert" not found Nov 02 21:02:33.104146 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.104120 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafaf75a-f79b-4c83-b57d-4a8a370f6519-openshift-service-ca\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.105880 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.105860 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-metrics-certs\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.160725 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.160655 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng7vr\" (UniqueName: \"kubernetes.io/projected/fafaf75a-f79b-4c83-b57d-4a8a370f6519-kube-api-access-ng7vr\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.608748 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.608710 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-maas-api-tls\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.611237 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.611216 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-maas-api-tls\") pod \"maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:33.827502 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:33.827464 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:34.198746 ip-10-0-130-111 kubenswrapper[5399]: W1102 21:02:34.198703 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafaf75a_f79b_4c83_b57d_4a8a370f6519.slice/crio-2d4691063d3ed834a04b74eaa49b1ea26642cdf0f705a788caf49b48f6d3bb3a WatchSource:0}: Error finding container 2d4691063d3ed834a04b74eaa49b1ea26642cdf0f705a788caf49b48f6d3bb3a: Status 404 returned error can't find the container with id 2d4691063d3ed834a04b74eaa49b1ea26642cdf0f705a788caf49b48f6d3bb3a Nov 02 21:02:34.199475 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:34.199449 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr"] Nov 02 21:02:34.374045 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:34.374011 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" event={"ID":"fafaf75a-f79b-4c83-b57d-4a8a370f6519","Type":"ContainerStarted","Data":"2d4691063d3ed834a04b74eaa49b1ea26642cdf0f705a788caf49b48f6d3bb3a"} Nov 02 21:02:35.380901 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:35.380861 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" event={"ID":"fafaf75a-f79b-4c83-b57d-4a8a370f6519","Type":"ContainerStarted","Data":"a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967"} Nov 02 21:02:35.381393 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:35.380989 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:35.495124 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:35.495021 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" podStartSLOduration=2.514298585 podStartE2EDuration="3.495006688s" totalImagesPullingTime="980.708103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:02:35.494462655 +0000 UTC m=+4198.105114748" watchObservedRunningTime="2026-11-02 21:02:35.495006688 +0000 UTC m=+4198.105658810" Nov 02 21:02:40.831675 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:40.831635 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:02:41.392189 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:02:41.392159 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:02:48.236663 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:02:48.236625 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:03:18.236352 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:03:18.236250 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:03:27.980336 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:27.980296 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr"] Nov 02 21:03:27.980787 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:27.980539 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" podUID="fafaf75a-f79b-4c83-b57d-4a8a370f6519" containerName="maas-api" containerID="cri-o://a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967" gracePeriod=30 Nov 02 21:03:28.233206 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.233143 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:03:28.355138 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.355103 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-maas-api-tls\") pod \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " Nov 02 21:03:28.355362 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.355194 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-metrics-certs\") pod \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " Nov 02 21:03:28.355362 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.355348 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fafaf75a-f79b-4c83-b57d-4a8a370f6519-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafaf75a-f79b-4c83-b57d-4a8a370f6519-openshift-service-ca\") pod \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " Nov 02 21:03:28.355498 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.355428 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fafaf75a-f79b-4c83-b57d-4a8a370f6519-kube-api-access-ng7vr\" (UniqueName: \"kubernetes.io/projected/fafaf75a-f79b-4c83-b57d-4a8a370f6519-kube-api-access-ng7vr\") pod \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\" (UID: \"fafaf75a-f79b-4c83-b57d-4a8a370f6519\") " Nov 02 21:03:28.356593 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.356550 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafaf75a-f79b-4c83-b57d-4a8a370f6519-openshift-service-ca" pod "fafaf75a-f79b-4c83-b57d-4a8a370f6519" (UID: "fafaf75a-f79b-4c83-b57d-4a8a370f6519"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:03:28.357627 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.357592 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafaf75a-f79b-4c83-b57d-4a8a370f6519-kube-api-access-ng7vr" pod "fafaf75a-f79b-4c83-b57d-4a8a370f6519" (UID: "fafaf75a-f79b-4c83-b57d-4a8a370f6519"). InnerVolumeSpecName "kube-api-access-ng7vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:03:28.357883 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.357847 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-metrics-certs" pod "fafaf75a-f79b-4c83-b57d-4a8a370f6519" (UID: "fafaf75a-f79b-4c83-b57d-4a8a370f6519"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:03:28.357883 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.357872 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-maas-api-tls" pod "fafaf75a-f79b-4c83-b57d-4a8a370f6519" (UID: "fafaf75a-f79b-4c83-b57d-4a8a370f6519"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:03:28.456717 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.456684 5399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-metrics-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:28.456717 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.456713 5399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fafaf75a-f79b-4c83-b57d-4a8a370f6519-openshift-service-ca\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:28.456983 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.456728 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng7vr\" (UniqueName: \"kubernetes.io/projected/fafaf75a-f79b-4c83-b57d-4a8a370f6519-kube-api-access-ng7vr\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:28.456983 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.456742 5399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fafaf75a-f79b-4c83-b57d-4a8a370f6519-maas-api-tls\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:28.669065 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.669026 5399 generic.go:356] "Generic (PLEG): container finished" podID="fafaf75a-f79b-4c83-b57d-4a8a370f6519" containerID="a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967" exitCode=0 Nov 02 21:03:28.669253 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.669081 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" Nov 02 21:03:28.669253 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.669108 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" event={"ID":"fafaf75a-f79b-4c83-b57d-4a8a370f6519","Type":"ContainerDied","Data":"a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967"} Nov 02 21:03:28.669253 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.669164 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr" event={"ID":"fafaf75a-f79b-4c83-b57d-4a8a370f6519","Type":"ContainerDied","Data":"2d4691063d3ed834a04b74eaa49b1ea26642cdf0f705a788caf49b48f6d3bb3a"} Nov 02 21:03:28.669253 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.669185 5399 scope.go:122] "RemoveContainer" containerID="a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967" Nov 02 21:03:28.679536 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.679516 5399 scope.go:122] "RemoveContainer" containerID="a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967" Nov 02 21:03:28.679859 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:03:28.679838 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967\": container with ID starting with a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967 not found: ID does not exist" containerID="a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967" Nov 02 21:03:28.679916 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.679869 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967"} err="failed to get container status \"a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967\": rpc error: code = NotFound desc = could not find container \"a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967\": container with ID starting with a2aba4256bab81580ef903e7f81104dbe131d408518f4f27f4c053fbade7c967 not found: ID does not exist" Nov 02 21:03:28.831122 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.831079 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr"] Nov 02 21:03:28.833175 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:28.833154 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3203ddac-5c44754f57-hhdtr"] Nov 02 21:03:29.506264 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:29.506229 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fafaf75a-f79b-4c83-b57d-4a8a370f6519" path="/var/lib/kubelet/pods/fafaf75a-f79b-4c83-b57d-4a8a370f6519/volumes" Nov 02 21:03:32.269091 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:32.269049 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:03:33.886602 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:33.886554 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln"] Nov 02 21:03:33.887137 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:33.887093 5399 cpu_manager.go:414] "containerMap: removing container" podUID="fafaf75a-f79b-4c83-b57d-4a8a370f6519" containerName="maas-api" Nov 02 21:03:33.887137 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:33.887106 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fafaf75a-f79b-4c83-b57d-4a8a370f6519" containerName="maas-api" Nov 02 21:03:33.887257 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:33.887177 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="fafaf75a-f79b-4c83-b57d-4a8a370f6519" containerName="maas-api" Nov 02 21:03:33.891722 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:33.891704 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:33.956478 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:33.956449 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-ac1065-openshift-default-dockercfg-6j8r8\"" Nov 02 21:03:33.996086 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:33.996049 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln"] Nov 02 21:03:34.014052 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.014014 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-envoy\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.014052 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.014054 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-token\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.014369 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.014074 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mfd\" (UniqueName: \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-kube-api-access-t5mfd\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.014369 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.014150 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-certs\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.014369 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.014184 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-podinfo\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.014369 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.014209 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-data\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.014617 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.014361 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-socket\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.014617 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.014431 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-credential-socket\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.014617 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.014473 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istiod-ca-cert\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115138 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115096 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-socket\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115385 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115155 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-credential-socket\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115385 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115182 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istiod-ca-cert\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115385 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115249 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-envoy\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115385 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115268 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-token\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115385 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115315 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mfd\" (UniqueName: \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-kube-api-access-t5mfd\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115385 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115334 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-certs\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115385 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115350 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-podinfo\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115814 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115612 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-socket\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115814 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115662 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-credential-socket\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115814 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115668 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-data\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115938 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115833 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-certs\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.115938 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.115887 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-data\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.116245 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.116221 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istiod-ca-cert\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.117598 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.117576 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-envoy\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.117867 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.117849 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-podinfo\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.173028 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.172943 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-token\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.173746 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.173723 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mfd\" (UniqueName: \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-kube-api-access-t5mfd\") pod \"e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.205363 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.205328 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:34.621107 ip-10-0-130-111 kubenswrapper[5399]: W1102 21:03:34.621068 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede73f76_a07c_4c97_a8c8_56d5fa1f23f3.slice/crio-ba9fbc9bae31f89333697388bf01a4a6cd1f5f95d1f1588c075130acb01789f6 WatchSource:0}: Error finding container ba9fbc9bae31f89333697388bf01a4a6cd1f5f95d1f1588c075130acb01789f6: Status 404 returned error can't find the container with id ba9fbc9bae31f89333697388bf01a4a6cd1f5f95d1f1588c075130acb01789f6 Nov 02 21:03:34.622157 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.622120 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln"] Nov 02 21:03:34.623249 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.623218 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:03:34.623332 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.623294 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:03:34.623332 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.623325 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:03:34.704616 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.704580 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" event={"ID":"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3","Type":"ContainerStarted","Data":"bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9"} Nov 02 21:03:34.704820 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.704648 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" event={"ID":"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3","Type":"ContainerStarted","Data":"ba9fbc9bae31f89333697388bf01a4a6cd1f5f95d1f1588c075130acb01789f6"} Nov 02 21:03:34.822117 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:34.822062 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" podStartSLOduration=1.822046495 podStartE2EDuration="1.822046495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:03:34.819760309 +0000 UTC m=+4257.430412414" watchObservedRunningTime="2026-11-02 21:03:34.822046495 +0000 UTC m=+4257.432698585" Nov 02 21:03:35.206093 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:35.206060 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:35.211049 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:35.211021 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:35.710305 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:35.710251 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:35.711316 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:35.711270 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:03:48.236988 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:03:48.236945 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:03:51.128671 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:03:51.128636 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:04:00.842506 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:00.842469 5399 generic.go:356] "Generic (PLEG): container finished" podID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerID="bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9" exitCode=0 Nov 02 21:04:00.842945 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:00.842543 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" event={"ID":"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3","Type":"ContainerDied","Data":"bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9"} Nov 02 21:04:00.842945 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:00.842942 5399 scope.go:122] "RemoveContainer" containerID="bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9" Nov 02 21:04:00.844097 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:00.844073 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:04:00.844169 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:00.844153 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:04:00.844211 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:00.844184 5399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:04:01.856143 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:01.856105 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" event={"ID":"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3","Type":"ContainerStarted","Data":"7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc"} Nov 02 21:04:02.206366 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:02.206245 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:04:02.207468 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:02.207441 5399 patch_prober.go:29] interesting pod/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:04:02.207598 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:02.207492 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:04:03.206624 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:03.206586 5399 patch_prober.go:29] interesting pod/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:04:03.207032 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:03.206640 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:04:04.206490 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:04.206457 5399 patch_prober.go:29] interesting pod/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:04:04.206684 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:04.206507 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:04:05.206143 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:05.206108 5399 patch_prober.go:29] interesting pod/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:04:05.206367 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:05.206170 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:04:05.710235 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:05.710197 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:04:06.206117 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:06.206073 5399 patch_prober.go:29] interesting pod/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Nov 02 21:04:06.206329 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:06.206132 5399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Nov 02 21:04:07.210452 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:07.210422 5399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:04:07.211384 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:07.211367 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:04:10.051411 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:10.051381 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2"] Nov 02 21:04:10.101335 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:10.101302 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57c93993-fpkp2"] Nov 02 21:04:11.505951 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:04:11.505919 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5731ea0-9ec6-4e21-97ba-4b45d3513952" path="/var/lib/kubelet/pods/d5731ea0-9ec6-4e21-97ba-4b45d3513952/volumes" Nov 02 21:04:18.236940 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:04:18.236897 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:04:48.236083 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:04:48.235997 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:05:18.235923 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:05:18.235881 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:05:39.872072 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:05:39.872037 5399 scope.go:122] "RemoveContainer" containerID="b999377020336e180e4307820f30da96f6fdfdb13cfc576a3a982ce3d98c356b" Nov 02 21:05:39.881301 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:05:39.881261 5399 scope.go:122] "RemoveContainer" containerID="469b8c33c4a5442a98ae07fc657902c902971920e445fbcab6264e3045b31bc0" Nov 02 21:05:48.235634 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:05:48.235592 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:05:51.744202 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:05:51.744162 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:05:55.827407 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:05:55.827370 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln"] Nov 02 21:05:55.827899 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:05:55.827659 5399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" containerID="cri-o://7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc" gracePeriod=30 Nov 02 21:06:01.080496 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.080469 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:06:01.152450 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152365 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-socket\") pod \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " Nov 02 21:06:01.152450 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152421 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-data\") pod \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " Nov 02 21:06:01.152450 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152452 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-certs\") pod \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " Nov 02 21:06:01.152702 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152503 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istiod-ca-cert\") pod \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " Nov 02 21:06:01.152702 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152565 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-credential-socket\") pod \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " Nov 02 21:06:01.152702 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152614 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-kube-api-access-t5mfd\" (UniqueName: \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-kube-api-access-t5mfd\") pod \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " Nov 02 21:06:01.152702 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152642 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-podinfo\") pod \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " Nov 02 21:06:01.152702 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152671 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-envoy\") pod \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " Nov 02 21:06:01.152977 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152705 5399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-token\" (UniqueName: \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-token\") pod \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\" (UID: \"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3\") " Nov 02 21:06:01.152977 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152701 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-socket" pod "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" (UID: "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:06:01.152977 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152767 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-data" pod "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" (UID: "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:06:01.152977 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152780 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-certs" pod "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" (UID: "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:06:01.152977 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152898 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-credential-socket" pod "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" (UID: "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:06:01.152977 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152946 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istiod-ca-cert" pod "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" (UID: "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:06:01.153305 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.152994 5399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-credential-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:01.153305 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.153013 5399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-socket\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:01.153305 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.153026 5399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-data\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:01.153305 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.153038 5399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-workload-certs\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:01.155084 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.155058 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-envoy" pod "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" (UID: "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:06:01.155084 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.155078 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-token" pod "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" (UID: "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:06:01.155407 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.155383 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-podinfo" pod "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" (UID: "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:06:01.155481 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.155415 5399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-kube-api-access-t5mfd" pod "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" (UID: "ede73f76-a07c-4c97-a8c8-56d5fa1f23f3"). InnerVolumeSpecName "kube-api-access-t5mfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:06:01.254549 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.254515 5399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-envoy\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:01.254549 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.254543 5399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-token\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:01.254549 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.254552 5399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istiod-ca-cert\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:01.254790 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.254563 5399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mfd\" (UniqueName: \"kubernetes.io/projected/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-kube-api-access-t5mfd\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:01.254790 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.254572 5399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3-istio-podinfo\") on node \"ip-10-0-130-111.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:06:01.509992 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.509915 5399 generic.go:356] "Generic (PLEG): container finished" podID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerID="7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc" exitCode=0 Nov 02 21:06:01.509992 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.509955 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" event={"ID":"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3","Type":"ContainerDied","Data":"7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc"} Nov 02 21:06:01.509992 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.509964 5399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" Nov 02 21:06:01.509992 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.509992 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln" event={"ID":"ede73f76-a07c-4c97-a8c8-56d5fa1f23f3","Type":"ContainerDied","Data":"ba9fbc9bae31f89333697388bf01a4a6cd1f5f95d1f1588c075130acb01789f6"} Nov 02 21:06:01.510348 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.510014 5399 scope.go:122] "RemoveContainer" containerID="7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc" Nov 02 21:06:01.522768 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.522383 5399 scope.go:122] "RemoveContainer" containerID="bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9" Nov 02 21:06:01.531077 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.531049 5399 scope.go:122] "RemoveContainer" containerID="7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc" Nov 02 21:06:01.531377 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:06:01.531353 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc\": container with ID starting with 7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc not found: ID does not exist" containerID="7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc" Nov 02 21:06:01.531454 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.531387 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc"} err="failed to get container status \"7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc\": rpc error: code = NotFound desc = could not find container \"7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc\": container with ID starting with 7db764143c4b9a33facaece22481467676fdf69d5dd698af61291610f70354bc not found: ID does not exist" Nov 02 21:06:01.531454 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.531405 5399 scope.go:122] "RemoveContainer" containerID="bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9" Nov 02 21:06:01.531647 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:06:01.531628 5399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9\": container with ID starting with bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9 not found: ID does not exist" containerID="bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9" Nov 02 21:06:01.531712 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.531656 5399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9"} err="failed to get container status \"bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9\": rpc error: code = NotFound desc = could not find container \"bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9\": container with ID starting with bc47c4a7125eb71846c47eec68d8ce780ab9a2463167e040f914218530f858a9 not found: ID does not exist" Nov 02 21:06:01.675378 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.675342 5399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln"] Nov 02 21:06:01.678447 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:01.678424 5399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ac1065-openshift-default-99cc59485-9ggln"] Nov 02 21:06:03.506684 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:03.506651 5399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" path="/var/lib/kubelet/pods/ede73f76-a07c-4c97-a8c8-56d5fa1f23f3/volumes" Nov 02 21:06:10.524984 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:10.524946 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:06:18.235837 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:06:18.235752 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:06:21.464252 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:21.464215 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:06:42.537098 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:06:42.537061 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:06:48.314151 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:06:48.314108 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:07:14.618616 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:14.618578 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vg6vn"] Nov 02 21:07:18.235503 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:07:18.235465 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:07:44.480418 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:44.480385 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lmtgk_4564bc57-50c6-41f8-9fb6-4c414641e45d/manager/0.log" Nov 02 21:07:45.556134 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:45.556097 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5ff67f6f5f-vrkd6_f14ae96b-1f8a-4b79-9984-d6340f921da1/manager/0.log" Nov 02 21:07:45.908962 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:45.908885 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vrfc2_7fa48662-147c-4804-abf3-737dc68d42d3/server/0.log" Nov 02 21:07:46.475609 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:46.475576 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4k4s4_b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01/proxy/0.log" Nov 02 21:07:47.189161 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:47.189081 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bc7c89d86-5dqt2_6d374d07-7e73-4478-b647-41f4ed60550a/maas-api/0.log" Nov 02 21:07:48.235792 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:07:48.235757 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:07:50.255548 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:50.255512 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5gnwm_e19d2022-b6cd-4cd2-90af-18082e986708/manager/0.log" Nov 02 21:07:50.761489 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:50.761457 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wbp8r_19e0ccc9-2f20-4852-a57f-78f4c48b9920/kuadrant-console-plugin/0.log" Nov 02 21:07:51.361863 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:51.361826 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vg6vn_3ef52526-da07-44fd-af13-1fa9a89feddf/limitador/0.log" Nov 02 21:07:52.740006 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:52.739975 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b88d44-mnwpp_ab733748-d93c-4b7b-96ab-b4fb1f74ea1c/kube-auth-proxy/0.log" Nov 02 21:07:53.581784 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:53.581747 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-f6jgj_c28a1f27-d0ed-412c-8dbb-17429467209a/istio-proxy/0.log" Nov 02 21:07:53.847531 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:53.847449 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-g7smq_9953c331-dda5-4f9e-b084-27e3b2caa37c/istio-proxy/0.log" Nov 02 21:07:55.179568 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:55.179531 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-zfbjs_2fc88c07-4a0d-4472-bf37-b35b17b58759/istio-proxy/0.log" Nov 02 21:07:56.913505 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:56.913473 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr_104f859d-9978-4548-91da-a1d3e1438dca/main/0.log" Nov 02 21:07:57.169079 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:57.169000 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-d5kgg_d2f7e6ae-694c-43dc-b392-a6414481c8cf/main/0.log" Nov 02 21:07:57.643238 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:57.643210 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-mkqnr_930a980d-dfca-4c6e-964f-38a7496267fe/main/0.log" Nov 02 21:07:58.105071 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:58.105040 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dxf74_7d6e94b5-d4a7-48d4-ab4b-27b457ed367c/main/0.log" Nov 02 21:07:58.356462 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:07:58.356382 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj_87db511f-cd55-456d-9b98-af4df0327cfc/main/0.log" Nov 02 21:08:18.237578 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:08:18.237539 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:08:18.745258 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:18.745224 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-lmtgk_4564bc57-50c6-41f8-9fb6-4c414641e45d/manager/0.log" Nov 02 21:08:19.539239 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:19.539208 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5ff67f6f5f-vrkd6_f14ae96b-1f8a-4b79-9984-d6340f921da1/manager/0.log" Nov 02 21:08:19.935643 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:19.935613 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vrfc2_7fa48662-147c-4804-abf3-737dc68d42d3/server/0.log" Nov 02 21:08:20.408549 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:20.408518 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4k4s4_b4e7cfc8-fd01-4be3-a2e4-f3c8c0bf1f01/proxy/0.log" Nov 02 21:08:20.950570 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:20.950539 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bc7c89d86-5dqt2_6d374d07-7e73-4478-b647-41f4ed60550a/maas-api/0.log" Nov 02 21:08:23.629468 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:23.629432 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5gnwm_e19d2022-b6cd-4cd2-90af-18082e986708/manager/0.log" Nov 02 21:08:24.149189 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:24.149161 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wbp8r_19e0ccc9-2f20-4852-a57f-78f4c48b9920/kuadrant-console-plugin/0.log" Nov 02 21:08:24.656977 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:24.656949 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vg6vn_3ef52526-da07-44fd-af13-1fa9a89feddf/limitador/0.log" Nov 02 21:08:25.853970 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:25.853941 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67b88d44-mnwpp_ab733748-d93c-4b7b-96ab-b4fb1f74ea1c/kube-auth-proxy/0.log" Nov 02 21:08:26.681066 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:26.681033 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-f6jgj_c28a1f27-d0ed-412c-8dbb-17429467209a/istio-proxy/0.log" Nov 02 21:08:26.937420 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:26.937338 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-g7smq_9953c331-dda5-4f9e-b084-27e3b2caa37c/istio-proxy/0.log" Nov 02 21:08:28.277486 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:28.277455 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-zfbjs_2fc88c07-4a0d-4472-bf37-b35b17b58759/istio-proxy/0.log" Nov 02 21:08:29.724702 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:29.724669 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-kdxkr_104f859d-9978-4548-91da-a1d3e1438dca/main/0.log" Nov 02 21:08:29.969545 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:29.969514 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-d5kgg_d2f7e6ae-694c-43dc-b392-a6414481c8cf/main/0.log" Nov 02 21:08:30.428126 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:30.428099 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-mkqnr_930a980d-dfca-4c6e-964f-38a7496267fe/main/0.log" Nov 02 21:08:30.937166 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:30.937133 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dxf74_7d6e94b5-d4a7-48d4-ab4b-27b457ed367c/main/0.log" Nov 02 21:08:31.197763 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:08:31.197685 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lmgvj_87db511f-cd55-456d-9b98-af4df0327cfc/main/0.log" Nov 02 21:08:48.237071 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:08:48.237030 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:09:18.236454 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:09:18.236361 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:09:40.530352 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:09:40.530322 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5ff67f6f5f-vrkd6_f14ae96b-1f8a-4b79-9984-d6340f921da1/manager/0.log" Nov 02 21:09:41.020811 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:09:41.020773 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5ff67f6f5f-vrkd6_f14ae96b-1f8a-4b79-9984-d6340f921da1/manager/0.log" Nov 02 21:09:48.235220 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:09:48.235179 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:10:18.238464 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:10:18.238421 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:10:28.797634 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:28.797601 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5ff67f6f5f-vrkd6_f14ae96b-1f8a-4b79-9984-d6340f921da1/manager/0.log" Nov 02 21:10:29.341759 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:29.341717 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5ff67f6f5f-vrkd6_f14ae96b-1f8a-4b79-9984-d6340f921da1/manager/0.log" Nov 02 21:10:40.746623 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:40.746594 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wkg2t_76aa7437-fa4a-42a6-90d5-9043d9b8a71d/global-pull-secret-syncer/0.log" Nov 02 21:10:43.361731 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:43.361698 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z28kx_4c6469de-dc6b-405f-b14f-cf765fd9d06f/konnectivity-agent/0.log" Nov 02 21:10:43.859567 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:43.859515 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-111.us-west-2.compute.internal_ee4b9ca33e3caae4f62b24687213e3f8/haproxy/0.log" Nov 02 21:10:48.235851 ip-10-0-130-111 kubenswrapper[5399]: E1102 21:10:48.235762 5399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pst9l" containerName="extractor" Nov 02 21:10:49.855870 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:49.855835 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5gnwm_e19d2022-b6cd-4cd2-90af-18082e986708/manager/0.log" Nov 02 21:10:49.964079 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:49.964046 5399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6"] Nov 02 21:10:49.964515 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:49.964502 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" Nov 02 21:10:49.964566 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:49.964516 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" Nov 02 21:10:49.964566 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:49.964526 5399 cpu_manager.go:414] "containerMap: removing container" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" Nov 02 21:10:49.964566 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:49.964531 5399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" Nov 02 21:10:49.964673 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:49.964589 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" Nov 02 21:10:49.964673 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:49.964599 5399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede73f76-a07c-4c97-a8c8-56d5fa1f23f3" containerName="istio-proxy" Nov 02 21:10:49.967763 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:49.967741 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.020048 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.020020 5399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zxnk7\"/\"default-dockercfg-m52f5\"" Nov 02 21:10:50.020694 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.020672 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6"] Nov 02 21:10:50.068391 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.068353 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxnk7\"/\"openshift-service-ca.crt\"" Nov 02 21:10:50.068579 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.068422 5399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxnk7\"/\"kube-root-ca.crt\"" Nov 02 21:10:50.087982 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.087942 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-sys\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.087982 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.087982 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px59v\" (UniqueName: \"kubernetes.io/projected/f5884d3e-2e52-4284-97d0-46669716bbe4-kube-api-access-px59v\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.088191 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.088163 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-lib-modules\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.088244 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.088191 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-proc\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.088244 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.088212 5399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-podres\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.189414 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.189334 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-lib-modules\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.189414 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.189369 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-proc\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.189414 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.189386 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-podres\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.189645 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.189425 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-sys\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.189645 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.189444 5399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px59v\" (UniqueName: \"kubernetes.io/projected/f5884d3e-2e52-4284-97d0-46669716bbe4-kube-api-access-px59v\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.189645 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.189463 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-lib-modules\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.189645 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.189463 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-proc\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.189645 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.189516 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-podres\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.189645 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.189526 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5884d3e-2e52-4284-97d0-46669716bbe4-sys\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.245745 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.245713 5399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px59v\" (UniqueName: \"kubernetes.io/projected/f5884d3e-2e52-4284-97d0-46669716bbe4-kube-api-access-px59v\") pod \"perf-node-gather-daemonset-62lv6\" (UID: \"f5884d3e-2e52-4284-97d0-46669716bbe4\") " pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.280662 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.280628 5399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:50.644311 ip-10-0-130-111 kubenswrapper[5399]: W1102 21:10:50.644258 5399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf5884d3e_2e52_4284_97d0_46669716bbe4.slice/crio-95ada8dd67a4f17d4f759ad9a76b260ff4bd893b96e6ace916f13061bb01f947 WatchSource:0}: Error finding container 95ada8dd67a4f17d4f759ad9a76b260ff4bd893b96e6ace916f13061bb01f947: Status 404 returned error can't find the container with id 95ada8dd67a4f17d4f759ad9a76b260ff4bd893b96e6ace916f13061bb01f947 Nov 02 21:10:50.646308 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.646264 5399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6"] Nov 02 21:10:50.665390 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:50.665369 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-wbp8r_19e0ccc9-2f20-4852-a57f-78f4c48b9920/kuadrant-console-plugin/0.log" Nov 02 21:10:51.084562 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:51.084515 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" event={"ID":"f5884d3e-2e52-4284-97d0-46669716bbe4","Type":"ContainerStarted","Data":"b0d23d963f281e05c60ad49801fc9c86d8ec374efdc79d48f3bf883130d6a4ea"} Nov 02 21:10:51.085106 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:51.084574 5399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" event={"ID":"f5884d3e-2e52-4284-97d0-46669716bbe4","Type":"ContainerStarted","Data":"95ada8dd67a4f17d4f759ad9a76b260ff4bd893b96e6ace916f13061bb01f947"} Nov 02 21:10:51.085106 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:51.084601 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:10:51.198541 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:51.198482 5399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" podStartSLOduration=2.198469228 podStartE2EDuration="2.198469228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:10:51.196686236 +0000 UTC m=+4693.807338331" watchObservedRunningTime="2026-11-02 21:10:51.198469228 +0000 UTC m=+4693.809121318" Nov 02 21:10:51.478082 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:51.477999 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vg6vn_3ef52526-da07-44fd-af13-1fa9a89feddf/limitador/0.log" Nov 02 21:10:54.461491 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:54.461460 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93c86b88-12f0-4433-b67b-7c97e7670cdf/alertmanager/0.log" Nov 02 21:10:54.864136 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:54.864105 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93c86b88-12f0-4433-b67b-7c97e7670cdf/config-reloader/0.log" Nov 02 21:10:55.264454 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:55.264367 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93c86b88-12f0-4433-b67b-7c97e7670cdf/kube-rbac-proxy-web/0.log" Nov 02 21:10:55.666579 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:55.666539 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93c86b88-12f0-4433-b67b-7c97e7670cdf/kube-rbac-proxy/0.log" Nov 02 21:10:56.068248 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:56.068220 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93c86b88-12f0-4433-b67b-7c97e7670cdf/kube-rbac-proxy-metric/0.log" Nov 02 21:10:56.470374 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:56.470268 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93c86b88-12f0-4433-b67b-7c97e7670cdf/prom-label-proxy/0.log" Nov 02 21:10:56.871271 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:56.871224 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93c86b88-12f0-4433-b67b-7c97e7670cdf/init-config-reloader/0.log" Nov 02 21:10:57.100719 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:10:57.100688 5399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zxnk7/perf-node-gather-daemonset-62lv6" Nov 02 21:11:02.122598 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:02.122571 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x4ttn_002bc527-a392-444a-85a5-2dbe005559f4/node-exporter/0.log" Nov 02 21:11:02.523156 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:02.523083 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x4ttn_002bc527-a392-444a-85a5-2dbe005559f4/kube-rbac-proxy/0.log" Nov 02 21:11:02.924521 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:02.924494 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x4ttn_002bc527-a392-444a-85a5-2dbe005559f4/init-textfile/0.log" Nov 02 21:11:04.560463 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:04.560423 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_503bac07-a421-4771-966e-51cd80f926f4/prometheus/0.log" Nov 02 21:11:04.963238 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:04.963141 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_503bac07-a421-4771-966e-51cd80f926f4/config-reloader/0.log" Nov 02 21:11:05.366427 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:05.366391 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_503bac07-a421-4771-966e-51cd80f926f4/thanos-sidecar/0.log" Nov 02 21:11:05.767436 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:05.767347 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_503bac07-a421-4771-966e-51cd80f926f4/kube-rbac-proxy-web/0.log" Nov 02 21:11:06.168567 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:06.168530 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_503bac07-a421-4771-966e-51cd80f926f4/kube-rbac-proxy/0.log" Nov 02 21:11:06.570541 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:06.570512 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_503bac07-a421-4771-966e-51cd80f926f4/kube-rbac-proxy-thanos/0.log" Nov 02 21:11:06.975318 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:06.975220 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_503bac07-a421-4771-966e-51cd80f926f4/init-config-reloader/0.log" Nov 02 21:11:07.388741 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:07.388715 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dbx7g_8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/prometheus-operator/0.log" Nov 02 21:11:07.790101 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:07.790027 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dbx7g_8a6c8be3-97cb-4ac9-be0e-b6bf3fff4a98/kube-rbac-proxy/0.log" Nov 02 21:11:08.606862 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:08.606834 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c566bd4cd-nbpxg_a3371f31-30e5-4c4c-9a08-e036e0b221b6/telemeter-client/0.log" Nov 02 21:11:09.009513 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:09.009443 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c566bd4cd-nbpxg_a3371f31-30e5-4c4c-9a08-e036e0b221b6/reload/0.log" Nov 02 21:11:09.410458 ip-10-0-130-111 kubenswrapper[5399]: I1102 21:11:09.410432 5399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c566bd4cd-nbpxg_a3371f31-30e5-4c4c-9a08-e036e0b221b6/kube-rbac-proxy/0.log"